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October 11, 2019 
Mr. Ali Bahrami 

Associate Administrator for Aviation Safety 
Federal Aviation Administration 
800 Independence Avenue SW 
Washington, DC 20591 

Dear Mr. Bahrami, 

On June 1, 2019, you chartered the Boeing 737 MAX Flight Control System Joint Authorities Technical 
Review (JATR), consisting of technical representatives from the FAA, National Aeronautics and Space 
Administration (NASA), and civil aviation authorities from Australia, Brazil, Canada, China, Europe, 
Indonesia, Japan, Singapore, and the United Arab Emirates. The members of the JATR team wish to 
thank you for the opportunity to conduct this review and to share our observations and findings. You also 
invited JATR members to submit recommendations whether or not they represented a consensus. Per your 
guidance, a compilation of those recommendations from JATR members is attached. It has been a 
privilege for us to work collaboratively on this multi-national team. 

In addition, our review of work conducted during the certification process would not have been possible 
without the notable support of a number of FAA aircraft certification, evaluation, and oversight 
personnel. Last but not least, we would like to thank Boeing for its cooperation in our review, including 
diverting people and other resources from their intense effort to return the airplane to service in order to 
respond to many of the issues that the JATR team raised. 

Overview . The FAA’s aircraft certification process has played a major role in producing airliners with an 
exemplary safety record consisting of a five-year worldwide average of only one fatal airliner crash for 
every 2 14 to 3 million flights, and a U.S. record of only one airline passenger fatality in more than 10 
years. Nonetheless, as with any system that is designed and operated by humans, the certification process 
can never be perfect, and the two tragic crashes that resulted in the creation of the JATR reveal a critical 
need to review the process to determine whether improvement and modernization are warranted. 

After extensive effort, the JATR members have made many recommendations regarding modernization 
and improvement of the certification process. Some of the recommendations are very broad in their 
application and others are more specific. 

Broad Recommendations . Some of the broader recommendations derive from the increasing complexity 
of aircraft systems, particularly automated systems and the interaction and the interrelationship between 
systems. As aircraft systems become more complex, ensuring that the certification process adequately 
addresses potential operational and safety ramifications for the entire aircraft that may be caused by the 
failure or inappropriate operation of any system on the aircraft becomes not only far more important, but 
also far more difficult. 
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Other broader recommendations raise the foundational issue of whether a process that has historically 
served the industry well for decades based largely upon compliance needs to be revisited to address not 
only compliance but also safety. As systems become more complex and may interact in unforeseeable 
ways, the likelihood increases that regulations and standards will not address every conceivable scenario. 
To the extent they do not address every scenario, compliance with every applicable regulation and 
standard does not necessarily ensure safety. Moreover, as systems become more complex, the certification 
process should ensure that aircraft incorporate fail-safe design principles. These principles prioritize the 
elimination or mitigation of hazards through design, minimizing reliance on pilot action as primary means 
of risk mitigation. 


Specific Recommendations . The specific recommendations include reviewing whether the ODA process 
can be made less cumbersome and bureaucratic to avoid stifling needed communications. The 
recommendations do not address the desirability of the ODA concept in general, but they do recommend 
examining how to help ensure adequate communications in future certification processes about important 
characteristics of what is being certificated. 

Query, for example, whether inadequate communications were partly responsible for the failure to 
address potential unintended consequences from the evolution of MCAS from a relatively benign system 
to a much more aggressive system; and query whether inadequate communications played a role in the 
failure to address potential unintended consequences that can result from designing software for one 
scenario - in this case, high-speed windup turns - and then modifying the software for a different scenario 
- in this case reducing the pitch-up tendency at higher angles of attack at low speeds. 

Other specific recommendations relate to revisiting the FAA’s standards regarding the time needed by 
pilots to identify and respond to problems that arise. Although existing standards have served the industry 
well for decades, the JATR members recommend an examination of whether those standards are as 
appropriate for the complex integrated systems in today’s airplanes. For example, when the failure or 
inappropriate operation of a system results in cascading failures and multiple alarms, query how 
adequately the certification process considers the impact of multiple alarms, along with possible startle 
effect, on the ability of pilots to respond appropriately. Inherent in this issue is the adequacy of training to 
help pilots be able to respond effectively to failures that they may never have encountered before, not 
even in training. 

Post-Certification . The initial scope of the JATR process was limited to the certification process itself, but 
the charter enabled the co-chairs, in their discretion, to expand the scope if warranted. Hence, some of the 
recommendations pertain to post-certification activities because of their potential to help improve the 
safety of future certification processes. 

Conclusion . The JATR members look forward to the FAA’s actions in response to these 
recommendations, and we hope they will help improve the certification process in ways that will continue 
to improve safety. 



Christopher A. Hart, Team Chair 
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Executive Summary 


Background 

In March 2017, the Federal Aviation Administration (FAA) issued an amended type certificate to 
The Boeing Company (Boeing) for the Boeing 737-8 MAX (B737-8 MAX), which was based on 
the type certificate for the Boeing 737 Next Generation (B737 NG). 1 In February 2018, the FAA 
approved the Boeing 737-9 MAX (B737-9 MAX). The B737-8 MAX and B737-9 MAX are 
hereinafter collectively referred to as the B737 MAX. 

The B737 MAX incorporated a number of design changes from the B737 NG. These changes 
included, but were not limited to, the incorporation of CFM LEAP-IB Series turbofan engines, 
structural changes to accommodate the new engines, advanced technology winglets, aft body 
aerodynamic improvements, fly-by-wire spoilers, and a Maneuvering Characteristics 
Augmentation System (MCAS) function. Collectively, the changes incorporated into the B737 
MAX design resulted in increased fuel efficiency, increased range, and a reduced noise profile 
compared to its predecessor, the B737 NG. 

On October 29, 2018, Lion AIR Flight JT610 (JT610), a B737-8 MAX, crashed shortly after 
takeoff in Jakarta, Indonesia. On November 7, 2018, the FAA issued Emergency Airworthiness 
Directive (AD) 2018-23-51 to require revising certificate limitations and operating procedures of 
the Airplane Flight Manual (AFM) for the B737 MAX to provide the flight crew with runaway 
horizontal stabilizer trim procedures to follow under certain conditions. 

On March 10, 2019, Ethiopian Airlines Flight 302 (ET302), also a B737-8 MAX, crashed shortly 
after takeoff in Addis Ababa, Ethiopia. On March 13, 2019, the FAA issued an emergency order 
prohibiting operation of the B737 MAX in the United States. 

Because of apparent similarities in factors that may have contributed to these accidents, the FAA 
Associate Administrator for Aviation Safety established a Joint Authorities Technical Review 
(JATR) to review the type certification of the flight control system on the B737 MAX. 

The JATR was chaired by Mr. Christopher Hart, an independent aviation safety professional and 
former Chairman of the National Transportation Safety Board (NTSB). The remainder of the 
JATR team was comprised of 28 members from the FAA, 2 the National Aeronautics and Space 
Administration (NASA), and nine civil aviation authorities (CAAs) representing: 

• Australia 

• Brazil 


1 The B737 MAX aircraft series is the fourth generation of the B737 (i.e., a “derivative” or “related” aircraft), 
succeeding the B737 NG. Boeing applied for certification of the B737-8 MAX (the first in the series) on June 30, 
2012, and the FAA certified the aircraft on March 8, 2017. See Type Certificate Data Sheet No. A16WE. 

2 The FAA participants selected for the JATR team did not participate in certification of the B737 MAX. 
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• Canada 

• China 

• European Union 

• Indonesia 

• Japan 

• Singapore 

• United Arab Emirates 

The FA A chartered the JATR to review the work conducted during the B737 MAX certification 
program, to assess whether compliance was shown with the required applicable airworthiness 
standards related to the flight control system and its interfaces, and to recommend improvements 
to the certification process if warranted. Of particular concern to the FAA in chartering the JATR 
was the function, evaluation, and certification of the MCAS function on the B737 MAX. 3 The 
JATR team’s review also focused on flight crew training and operational suitability of the 
design. The JATR team considered whether the appropriate regulations and policy were applied, 
as well as how applicable regulations and policy material could be improved to enhance safety. 

The FAA did not charter the JATR to review the entire certification process for all aspects of the 
aircraft, nor did it task the team to review details related to returning the B737 MAX to service. 
The FAA made clear that it did not create the JATR to inform its decision on returning the B737 
MAX to service. 

The JATR team conducted its review from approximately May through September 2019. The 
team met in person three times during this period for a total of four weeks and exchanged 
information electronically between meetings. The team received briefings from FAA and Boeing 
personnel knowledgeable of the B737 MAX program, and the team conducted an intensive two- 
week review of certification compliance documentation and related data held by Boeing. The 
JATR team also reviewed applicable FAA regulations and guidance. The team’s findings and 
compilation of members’ recommendations are commensurate with the infonnation made 
available to the team and with the time constraints inherent in such a review. 

The charter did not require consensus recommendations; the recommendations provided in this 
submittal are a compilation of team members’ recommendations. Also in accordance with the 
charter, the JATR team produced observations and findings but did not prepare a report. The 
team endeavored to provide sufficient context through the background information included with 
each of the 12 main recommendations below and through the detailed supporting observations, 
findings, and recommendations that follow this Executive Summary. 


3 The MCAS function resides in the aircraft’s flight control computer. 
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Summary of JATR Team Members’ Recommendations 
The Certification Process 
1. Changed Product Rule 

The FAA continually amends aircraft design regulations to improve safety. New aircraft designs 
are required to meet the latest amendments to the regulations, but in some circumstances, 
changes to previously approved designs can be certified under previous regulatory amendments. 
The process for determining the applicable amendments is governed by Section 21.101 of Title 
14 of the Code of Federal Regulations (CFR), known as the Changed Product Rule. 4 The design 
regulations for the B737 MAX included a combination of the following: 

• Regulatory amendments in effect when the B737 was originally certified in 1967. 

• Regulatory amendments in effect when Boeing applied for certification of the B737 
MAX project. 

• Regulatory amendments in effect during the time between original certification in 1967 
and application for certification of the B737 MAX. 

• Certain regulatory amendments that became effective after Boeing’s application date that 
the company elected to comply with. 5 

• Special conditions, exemptions, and equivalent level of safety findings (typical of similar 
certification projects). 

• Additional design requirements and conditions (ADRCs). 


4 The certification procedures for aircraft are in 14 CFR part 21. Subparts A through E specify certain regulations 
and the applicable airworthiness standards for type certification of new and changed products. Airworthiness 
standards for transport category aircraft are in 14 CFR parts 25 and 26. The term “changed product” includes 
changes that are made through an amended type certificate (ATC), a supplemental type certificate (STC), or an 
amended STC. Guidance for complying with the Changed Product Rule (14 CFR 21.101) is found in Advisory 
Circular 21.101-1B, Establishing the Certification Basis of Changed Aeronautical Products , and FAA Order 
8110.48A, How to Establish the Certification Basis for Changed Aeronautical Products. 

5 An applicant for a change to a type certificate must show that the change and areas affected by the change comply 
with the applicable airworthiness requirements in effect on the date of the application for the change (i.e., the latest 
amendment of the regulation), unless the applicant shows that the change meets the criteria for an exception set out 
in § 21.101(b) or (c). Under § 21.101(b), an applicant may propose a certification basis using an airworthiness 
requirement in effect before the date of application (but not earlier than the existing certification basis) if the earlier 
amendment is considered adequate and meets one of the criteria in § 21.101(b) - i.e., the change is not significant 
(§ 21.101(b)(1)); the area, system, component, equipment, or appliance is not affected by the change 

(§ 21.101(b)(2)); or compliance with latest amendment would not contribute materially to the level of safety of the 
product or would be impractical (§ 21.101(b)(3)). Even if an exception is available under § 21.101(b), an applicant 
may still elect to comply with the latest amendment. 


Ill 
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The Changed Product Rule requires changed areas of the design and areas affected by the change 
to be assessed for compliance, but allows unaffected areas of the aircraft not to be reassessed. 

The JATR team reviewed how the Changed Product Rule process was applied to the certification 
of the flight control system of the B737 MAX. The JATR team detennined that the Changed 
Product Rule process was followed and that the process was effective for addressing discrete 
changes. However, the team determined that the process did not adequately address cumulative 
effects, system integration, and human factors issues. The Changed Product Rule process allows 
the applicant 6 to only address in a limited way changed aspects (and areas affected by the 
change) and does not require analysis of all interactions at the aircraft level. 

The current Changed Product Rule process lacks an adequate assessment of how proposed 
design changes integrate with existing systems and the associated impact of this interaction at the 
aircraft level. A more fulsome assessment process would apply to establishing the certification 
basis as well as to finding compliance throughout the certification process. 

Recommendation R1 

Based on the JATR team’s observations and findings related to the application of the 
Changed Product Rule to the certification of the flight control system of the B737 MAX, 
JATR team members recommend that the FAA work with other civil aviation authorities to 
revise the harmonized approach to the certification of changed products. Changed Product 
Rules (e.g., 14 CFR §§ 21.19 & 21.101) and associated guidance (e.g., Advisory Circular 
21.101-1B and FAA Orders 8110.4C and 8110.48A) should be revised to require a top-down 
approach whereby every change is evaluated from an integrated whole aircraft system 
perspective. These revisions should include criteria for determining when core attributes of 
an existing transport category aircraft design make it incapable of supporting the safety 
advancements introduced by the latest regulations and should drive a design change or a 
need for a new type certificate. The aircraft system includes the aircraft itself with all its 
subsystems, the flight crew, and the maintenance crew. 

These Changed Product Rule revisions should take into consideration the following key 
principles: 

• A comprehensive integrated system-level analysis recognizing that in this complex 
interactive system, every change could interact with other parts of the system. 

• The assessment of proposed design changes on existing systems at the aircraft level 
includes using development assurance principles, system safety principles, and 


6 The term “applicants” as used in this document refers to persons applying for a type certificate (TC), a 
supplemental type certificate (STC), or an amendment to either a TC or STC. This includes both aircraft 
manufacturers (often referred to as original equipment manufacturers) and, in the case of STCs and amended STCs, 
aircraft modifiers as well. 
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validation & verification techniques. The level of assessment should be proportional 
to the impact of the change at the aircraft level. 

• The consideration of training and qualification offlight and maintenance personnel, 
as well as detailed explicit procedures for the safe operation of the aircraft. 


2. Development and use of up-to-date requirements and practices 

The JATR team reviewed the regulations, policy, and compliance methods applied to the B737 
MAX. The JATR team determined that some regulations, policies, and compliance methods that 
address safety issues related to system integration and human factors and that were available at 
the time of the B737 MAX certification process were not applied to the B737 MAX or were only 
partially applied in a way that failed to achieve the full safety benefit. In some cases, this failure 
to achieve the full safety benefit associated with the application of the latest compliance methods 
was because the FAA regulations and guidance were out of date. Another area the JATR team 
detennined is in need of an update is the guidance concerning pilot recognition time and pilot 
reaction time to failures. Additionally, the JATR team determined that new and novel application 
of specific design features was not adequately considered. 


Recommendation R2 

Based on the JATR team’s observations and findings related to the regulations, policy, and 
compliance methods applied to the B737MAX, JATR team members recommend that the 
FAA update regulations and guidance that are out of date and update certification 
procedures to ensure that the applied requirements, issue papers, means of compliance, and 
policies fully address the safety issues related to state-of-the-art designs employed on new 
projects. JATR team members also recommend that the FAA review its processes to ensure 
that regulations and guidance materials are kept up to date. 


3. Consistent interpretation and application of requirements 

The JATR team reviewed the certification of the Boeing B737 MAX flight control system and 
related interfaces to assess whether compliance was shown to the applicable design standards 
and requirements. The JATR team identified concerns with the consistent application and 
interpretation of regulatory guidance pertaining to the system safety assessment (SSA), handling 
qualities rating method (HQRM), and confonnity requirements for engineering simulators and 
devices. The application of 14 CFR § 25.1329 (Flight Guidance System) and § 25.1581 
(Airplane Flight Manual - General) and the supporting data and techniques used for § 25.201 
(Stall Demonstration) were questioned. 
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Recommendation R3 

Based on the JATR team’s observations and findings related to the certification of the B 73 7 
MAXflight control system and related interfaces, JATR team members recommend that the 
FAA review the B737 MAX compliance to 14 CFR §§ 25.1329 (Flight Guidance System), 
25.1581 (Airplane Flight Manual - General), and 25.201 (Stall Demonstration) and ensure 
the consistent application and interpretation of regulatory guidance material for the system 
safety assessment, handling qualities rating method, and conformity requirements for 
engineering simulators and devices. Should there be a non-compliance, the root cause 
should be identified and measures implemented to prevent recurrence. 


4 . Changes during the certification process 

The JATR team reviewed the type certification process (per FAA Order 8110.4C, Type 
Certification, and related guidance) to determine whether the process includes sufficient 
feedback paths to accommodate changes in aircraft design and methods of compliance during the 
lengthy span (e.g., five years) of a typical major aircraft certification program such as the B737 
MAX. The JATR team identified specific areas related to the evolution of the design of the 
MCAS where the certification deliverables were not updated during the certification program to 
reflect the changes to this function within the flight control system. In addition, the design 
assumptions were not adequately reviewed, updated, or validated; possible flight deck effects 
were not evaluated; the SSA and functional hazard assessment (FHA) were not consistently 
updated; and potential crew workload effects resulting from MCAS design changes were not 
identified. 


Recommendation R4 

Based on the JATR team’s observations and findings related to the FAA type certification 
process, JATR team members recommend that the FAA review and update the regulatory 
guidance pertaining to the type certification process with particular emphasis on early FAA 
involvement to ensure the FAA is aware of all design assumptions, the aircraft design, and 
all changes to the design in cases where a changed product process is used. The FAA 
should consider adding feedback paths in the process to ensure that compliance, system 
safety, and flight deck/human factors aspects are considered for the aircraft design 
throughout its development and certification. 
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5. Delegation of certification authority 

Under the FAA’s Organization Designation Authorization (ODA) program, the FAA granted 
Boeing designee authority over parts of the certification project. 7 FAA oversight of the 
certification process was performed by the FAA’s Boeing Aviation Safety Oversight Office 
(BASOO). 8 

The act of delegating, i.e., designating industry as representatives of a CAA, is well established 
and is common practice by the majority of CAAs around the world. Delegation provides CAAs 
with a pool of expertise to exercise approvals and findings of compliance within the scope of 
delegated authority. However, the ongoing oversight of designees and ODAs by the CAA is 
critically important to provide assurance to the CAA that safety and certification work is being 
carried out satisfactorily. 

The BASOO is required to perform a certification function, including making findings of 
compliance of retained (non-delegated) requirements, while also performing the oversight 
function of the Boeing ODA. The BASOO must have the resources to carry out these two 
primary functions without compromise. The JATR team concluded that FAA resource shortfalls 
in the BASOO (and other allocated resources) may have contributed to an inadequate number of 
FAA specialists being involved in the B737 MAX certification program. In some cases, BASOO 
engineers had limited experience and knowledge of key technical aspects of the B737 MAX 
program. 

The BASOO delegated a high percentage of approvals and findings of compliance to the Boeing 
ODA for the B737 MAX program. With adequate FAA engagement and oversight, the extent of 
delegation does not in itself compromise safety. However, in the B737 MAX program, the FAA 
had inadequate awareness of the MCAS function which, coupled with limited involvement, 
resulted in an inability of the FAA to provide an independent assessment of the adequacy of the 
Boeing proposed certification activities associated with MCAS. In addition, signs were reported 
of undue pressures on Boeing ODA engineering unit members (E-UMs) performing certification 
activities on the B737 MAX program, which further erodes the level of assurance in this system 
of delegation. 


7 Under 49 U.S.C. 44702(d), the FAA may delegate to a qualified private person a matter related to issuing 
certificates or related to the examination, testing, and inspection necessary to issue a certificate. 

8 FAA’s Transport Aircraft Seattle Aircraft Evaluation Group coordinated and assisted in the certification process. 
FAA also formed a Flight Standardization Board to evaluate and validate Boeing’s (as the applicant) proposed 
training program for the B737 MAX. 
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Recommendation R5 

Based on the JATR team’s observations and findings related to FAA ’s oversight by the Boeing 
Aviation Safety Oversight Office (BASOO), JATR team members recommend that the FAA 
conduct a workforce review of the BASOO engineer staffing level to ensure there is a 
sufficient number of experienced specialists to adequately perform certification and oversight 
duties, commensurate with the extent of work being performed by Boeing. The workforce 
levels should be such that decisions to retain responsibility for finding compliance are not 
constrained by a lack of experienced engineers. 

The FAA should review the Boeing Organization Designation Authorization (ODA) work 
environment and ODA manual to ensure the Boeing ODA engineering unit members (E-UMs) 
are working without any undue pressure when they are making decisions on behalf of the 
FAA. This review should include ensuring the E-UMs have open lines of communication to 
FAA certification engineers without fear ofpunitive action or process violation. 


Integrated Approach to Development and Certification 
6. Holistic, integrated aircraft-level approach 

The JATR team reviewed the design process of the flight control system and the related SSAs for 
the B737 MAX to assess whether the flight control system complies with applicable system 
design and safety requirements and standards. The JATR team found that the MCAS was not 
evaluated as a complete and integrated function in the certification documents that were 
submitted to the FAA. The lack of a unified top-down development and evaluation of the system 
function and its safety analyses, combined with the extensive and fragmented documentation, 
made it difficult to assess whether compliance was fully demonstrated. The MCAS design was 
based on data, architecture, and assumptions that were reused from a previous aircraft 
configuration without sufficient detailed aircraft-level evaluation of the appropriateness of such 
reuse, and without additional safety margins and features to address conditions, omissions, or 
errors not foreseen in the analyses. 


Recommendation R6 

Based on the JATR team’s observations and findings related to the design process of the 
flight control system and the related system safety assessments for the B 737 MAX, JATR 
team members recommend that the FAA promote a safety culture that drives a primary 
focus on the creation of safe products, which in turn comply with certification requirements. 
Aircraft functions should be assessed, not in an incremental and fragmented manner, but 
holistically at the aircraft level. System function and performance, including the effects of 
failures, should be demonstrated and associated assumptions should be challenged to 
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ensure robust designs are realized. The safety analysis process should be integrated with the 
aircraft development assurance process to ensure all safety requirements and associated 
assumptions are correct, complete, and verified. The FAA should encourage applicants to 
have a system safety function that is independent from the design organization, with the 
authority to impartially assess aircraft safety and influence the aircraft/system design 
details. Adoption of a safety management system is one way this can be achieved. 


1 . Human factors 

Humans design, build, maintain, and operate every part of the global aviation system. The 
enviable safety record of the aviation system is a direct result of human capabilities. At the same 
time, all aviation accidents are the result of human limitations. This is not to say that all 
accidents are the result of human error, but of human limitations, such as limitations to people’s 
imagination and their ability to foresee, predict, and anticipate possible situations. As the 
technology becomes more advanced, and as the operational environment becomes more 
complex, understanding the scope and nature of the interactions between the technology, the 
human, and the environment becomes more critical to aviation safety. This criticality of human 
factors to aviation safety has been recognized and has been codified in various rules such as 14 
CFR §§ 25.1302 (Installed Systems and Equipment for Use by the Flightcrew), 25.1309 
(Equipment, Systems, and Installations), and 25.1322 (Flightcrew Alerting). While issues in 
human-machine interaction are at the core of all recent aviation accidents and are implicated in 
the two B737 MAX accidents, the FAA has very few human factors and human system 
integration experts on its certification staff. The JATR team identified multiple human factors- 
related issues in the certification process. Because human factors is a cross-cutting aspect, related 
recommendations are made under several of the different areas identified in this summary. 

Recommendation R 7 

Based on the JATR team’s observations and findings related to human factors-related 
issues in the certification process, JATR team members recommend that the FAA integrate 
and emphasize human factors and human system integration throughout its certification 
process. Human factors-relevant policies and guidance should be expanded and clarified, 
and compliance with such regulatory requirements as 14 CFR §§ 25.1302 (Installed 
Systems and Equipment for Use by the Flightcrew), 25.1309 (Equipment, Systems, and 
Installations), and 25.1322 (Flightcrew Alerting) should be thoroughly verified and 
documented. To enable the thorough analysis and verification of compliance, the FAA 
should expand its aircraft certification resources in human factors and in human system 
integration. 
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8. Development assurance 

Development assurance is a methodology applied to aircraft and aircraft systems to ensure safe 
and compliant designs in increasingly complex and integrated aircraft systems. Design and 
analysis techniques traditionally applied to detenninistic risks or to conventional, non-complex 
systems may not provide adequate safety coverage for complex systems. As evidenced in the 
B737 MAX, integrated aircraft-level functions, such as the MCAS, present a risk of development 
error (requirements determination and design errors) and undesirable, unintended effects. 

The systematic use of development assurance techniques increases confidence that errors in 
requirements or design, and integration or interaction effects, have been adequately identified 
and corrected by the applicant. 

Boeing elected to meet the objectives of SAE International’s Aerospace Recommended Practice 
4754A, Guidelines for Development of Civil Aircraft and Systems, (ARP4754A) for development 
assurance of the B737 MAX. Issue Paper SA-1 documented the methods and means that Boeing 
used to show that the processes used for B737 MAX systems development were, when 
appropriate, in accordance with the objectives of ARP4754A and were an acceptable means of 
addressing requirement, design, and implementation errors. The use of ARP4754A is consistent 
with the guidance contained in FAA Advisory Circular (AC) 20-174, Development of Civil 
Aircraft and Systems. The JATR team identified areas where the Boeing processes can be 
improved to more robustly meet the development assurance objectives of ARP4754A. 

Recommendation R8 

Based on the JATR team ’s observations and findings related to the development assurance 
process applied to the design of the flight control system of the B737 MAX, JATR team 
members recommend that the FAA ensure applicants apply industry best practice for 
development assurance, including requirements management, visibility of assumptions, 
process assurance activities, and configuration management. The FAA should ensure 
achievement of the close coupling that is required between the applicant safety analysis 
process and the development assurance process to classify failure conditions and derive the 
level of rigor of design development and verification. A current example of industry best 
practice is SAE InternationaVs Aerospace Recommended Practice 4754A (ARP4754A). 

The FAA should review and amend Advisory Circular 20-174 to clearly articulate the 
principles ofARP4754A, promoting industry best practice for development assurance of 
aircraft and aircraft systems to address applicants’ design trend of increasing integration 
between aircraft functions and systems. 
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Impact of Design Changes on Operations and Training 

9. Impact of product design changes on operations 

A review of preliminary accident reports KNKT. 18.10.35.04 (for Lion AIR Flight JT610) and 
AI-0/19 (for Ethiopian Airlines Flight 302) indicates that the complex operational environment 
that faced the flight crews during the events leading up to the accidents and the associated flight 
crew workload may not have been anticipated in the certification process. Applicants make 
various assumptions during the requirements definition phase which can influence the design and 
how that design is certified. Boeing made several assumptions for the B737 MAX which directly 
influenced the design and certification of MCAS. The evaluation of an applicant’s operational 
design assumptions concerning crew response is under the purview of the FAA Aircraft 
Evaluation Group (AEG). 

FAA Order 8110.4C articulates the need for AEG’s early involvement in the certification 
process, starting at the requirements definition phase. Test pilots working in the certification 
process may not have complete knowledge of operational issues, while pilots working in the 
operational evaluation process may not have complete knowledge of certification issues. This 
gap may contribute to limited communication between the two processes, creating the potential 
for a lack of operational insight into the certification process. 

Recommendation R9 

Based on the JATR team’s findings and observations related to the operational design 
assumptions of crew response applied during the certification process for the flight control 
system of the B 73 7 MAX, JATR team members recommend that the FAA require the 
integration of certification and operational functions during the certification process. The 
FAA should be provided all system differences between related aircraft in order to 
adequately evaluate operational impact, systems integration, and human performance. 


10. Impact of product design changes on flight crew training 

A review of preliminary accident reports KNKT. 18.10.35.04 and AI-0/19 indicates that both 
flights suffered an extreme mis-trim event which involved the activation of the MCAS function. 
During the certification process, a decision was made to remove information relating to MCAS 
functionality from the draft Flight Crew Operating Manual (FCOM). This decision meant that 
the FAA Flight Standardization Board (FSB) was not fully aware of the MCAS function and was 
not in a position to adequately assess training needs. 

The Boeing AFM does not include ah the normal, non-normal, and emergency operating 
procedures as required by regulations. Boeing has included most of the operating procedures in 
the FCOM in accordance with FAA guidance (AC 25.1581-1 , Airplane Flight Manual). This 
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difference between the rule and the guidance enabled Boeing to make changes to aircraft 
operating procedures (via the FCOM) without requiring FAA approval for such changes. This 
can result in situations where the FAA is unaware of changes to nonnal, non-normal, and 
emergency operating procedures. Systems information is often included in the FCOM and may 
not be required to be included in the FAA-approved AFM. However, technology, even if it 
functions without pilot involvement, may be integrated with other aircraft systems, such that one 
system or functional failure could impact other systems and require pilot intervention. When 
such technologies, systems, or possible malfunctions are not included in the FCOM or in the 
Flight Crew Training Manual (FCTM), the pilot is unlikely to be aware of them and may fail to 
recognize a malfunction when it occurs or may not know how to respond appropriately. 

To be compliant with FAA regulations and guidance material, Boeing utilized four fundamental 
assumptions on crew actions in the flight control functional hazard assessment (FHA) for the 
B737 MAX and other Boeing models. The third assumption, taken from AC 25-7C, stated: “The 
pilot will take immediate action to reduce or eliminate high control forces by re-trimming or 
changing configuration or flight conditions.” 9 It is evident from the accident flights that the flight 
crews’ actions were not consistent with Boeing’s third operational design assumption. 

Recommendation RIO 

Based on the JATR team ’s findings and observations related to flight crew training, JATR 
team members recommend that the FAA require a documented process to determine what 
information will be included in the Airplane Flight Manual, the Flight Crew Operating 
Manual, and the Flight Crew Training Manual. The FAA should review training programs 
to ensure flight crews are competent in the handling of mis-trim events. 


11. Impact of product design changes on maintenance training 

The JATR team was tasked to consider maintenance suitability of the design. Due to lack of 
maintenance expertise on the JATR, the team was unable to make a detennination of such 
adequacy. 


9 Some of the FAA advisory circulars (ACs) referenced in this document are referred to at different revision levels 
depending on the context. Where the JATR team made an observation or finding about an AC as applied to the 
certification of the B737 MAX, the revision of the AC in effect at the time of certification was the version reviewed 
by the JATR team and is referenced accordingly. If such an AC has since been revised, the later revision of the AC 
is referred to in observations or findings about current content and in recommendations for improvements to FAA 
guidance. For example, several references are made to AC 25-7C, which was current at the time of the B737 MAX 
certification program and was used as compliance guidance. Other references are made to AC 25-7D, which is the 
current version of the AC and is therefore referred to in observations and findings about current content and in the 
JATR team’s recommendations for future enhancements to the AC. 
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Recommendation Rll 

JATR team members recommend that the FAA conduct a study to determine the adequacy 
of policy, guidance, and assumptions related to maintenance and ground handling training 
requirements. 


Post-Certification Activities 

12. Post-certification corrective actions and data sharing 

In accordance with its charter, the JATR team focused on reviewing the flight controls work 
conducted by Boeing and the FAA leading up to certification of the B737 MAX. The team did 
not conduct an in-depth review of post-certification activities, as this was beyond the initial 
scope of the JATR. However, during the course of its review, the team became aware of some 
aspects of post-certification activities. As a result, the team took advantage of the provision in 
the charter for expanding the JATR’s scope at the discretion of the Chair and Co-chair in order to 
identify additional observations and recommendations that have the potential for further 
enhancing aviation safety. 

Recommendation R12 

JATR team members recommend that the FAA review its policies for analyzing safety risk 
and implementing interim airworthiness directive action following a fatal transport aircraft 
accident. The FAA should ensure that it shares post-accident safety information with the 
international community to the maximum extent possible. 
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JATR Team Observations and Findings; Member Recommendations 


Terminology 

This submittal consists of observations and findings developed by the JATR team, as well as a 
compilation of team members’ recommendations, as follows: 

Observation 

An observation is a noteworthy fact or issue gained from the JATR team’s review of the FAA’s 
certification of the B737 MAX flight control system and its related interfaces. 

Finding 

A finding is a conclusion drawn by the JATR team based on review of design details, analyses, 
reports, or other factual evidence. 

Recommendation 

A recommendation is a proposed action for the FAA to consider and is intended to identify 
“what” is to be done, as opposed to “how” actions are to be accomplished. Recommendations are 
based on the JATR team’s findings and observations. Note that not all findings or observations 
necessarily resulted in a recommendation. 

Acronyms 

The JATR uses the following acronyms in this submittal. 

AC - Advisory Circular 
AD - Airworthiness Directive 

ADRC - Additional design requirements and conditions 
AEG - Aircraft Evaluation Group (FAA) 

AFM - Airplane Flight Manual 
AMC - Acceptable means of compliance 
AOA - Angle of attack 

ARAC - Aviation Rulemaking Advisory Committee 
ATC - Amended type certificate 

BASOO - Boeing Aviation Safety Oversight Office (FAA) 
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CAA - Civil Aviation Authority 

CFR - Code of Federal Regulations 

DOORS - Dynamic Object-Oriented Requirements System 

EASA - European Aviation Safety Agency 

EFS - Elevator feel shift 

E-UM - Engineering unit member 

FAA - Federal Aviation Administration 

FCC - Flight control computer 

FCOM - Flight Crew Operating Manual 

FCTM - Flight Crew Training Manual 

FHA - Functional hazard assessment 

FSB - Flight Standardization Board (FAA) 

HQRM - Handling qualities rating method 
HUD - Head-up display 

ICAO - International Civil Aviation Organization 

JATR - Joint Authorities Technical Review 

MCAS - Maneuvering Characteristics Augmentation System 

MMEL - Master minimum equipment list 

MSAD - Monitor Safety/Analyze Data 

NASA - National Aeronautics and Space Administration 

NTSB - National Transportation Safety Board 

ODA - Organization Designation Authorization 

OFE - Operational flight envelope 

PSSA - Preliminary system safety assessment 

SDAHWG - Systems Design and Analysis Harmoni z ation Working Group 
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SACO - Seattle Aircraft Certification Office 

S&MF - Single & multiple failure 

SSA - System safety assessment 

STC - Supplemental type certificate 

STS - Speed trim system 

TC - Type certificate 

TCDS -Type certificate data sheet 

FAA Guidance, Directives, and Regulations 

The JATR team also addresses the following FAA advisory circulars (ACs), FAA orders, and 
sections of Title 14 of the Code of Federal Regulations in this submittal. 

Note: Latest revisions of relevant FAA ACs and orders are listed below for reference in the 
context of recommended improvements that follow. However, the JATR team generally 
considered the revision of an FAA AC or order that was applicable during the certification of the 
B737 MAX, which in some cases was an earlier revision than listed below. 

FAA Advisory Circulars 

AC 20-174, Development of Civil Aircraft and Systems 

AC 21.101-1B, Establishing the Certification Basis of Changed Aeronautical Products 
AC 25-7D, Flight Test Guide for Certification of Transport Category Airplanes 
AC 25-1 IB, Electronic Flight Displays 

AC 25.1302-1, Installed Systems and Equipment for Use by the Flightcrew 
AC 25.1309-1 A, System Design and Analysis 
AC 25.1329-1C, Approval of Flight Guidance Systems 
AC 25.1581-1, Airplane Flight Manual 

AC 120-53B, Guidance for Conducting and Use of Flight Standardization Board Evaluations 
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FAA Orders 

FAA Order 8100.15B, Organization Designation Authorization Procedures 
FAA Order 8110.4C, Type Certification 

FAA Order 8110.48 A, How to Establish the Certification Basis for Changed Aeronautical 
Products 

FAA Order 8110.107A, Monitor Safety/Analyze Data 

Sections of Title 14 of the Code of Federal Regulations (CFR) 

PART 21 - CERTIFICATION PROCEDURES FOR PRODUCTS AND ARTICLES 

Subpart B - Type Certificates 

§21.19 Changes requiring a new type certificate 

Subpart D - Changes to Type Certificates 

§ 21.101 (Designation of applicable regulations) 

PART 25 - AIRWORTHINESS STANDARDS: TRANSPORT CATEGORY AIRPLANES 

Subpart B - Flight 

Stalls 

§ 25.201 Stall demonstration 
Miscellaneous Flight Requirements 
§ 25.255 Out-of-trim characteristics 
Subpart D - Design and Construction 
Control Systems 
§ 25.671 General 

§ 25.672 Stability augmentation and automatic and power-operated systems 

Subpart F - Equipment 

General 

§ 25.1302 Installed systems and equipment for use by the flightcrew 
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§ 25.1309 Equipment, systems, and installations 

Instruments: Installation 

§ 25.1322 Flightcrew alerting 

§ 25.1329 Flight guidance system 

Subpart G - Operating Limitations and Information 

Airplane Flight Manual 

§ 25.1581 General 

§ 25.1583 Operating limitations 

§ 25.1585 Operating procedures 

§ 25.1587 Performance information 

Observations, Findings, and Recommendations 

The following observations, Endings, and recommendations are based on the data and 
information that was made accessible to the JATR team. The team’s access to some information 
was limited by factors such as U.S. export controls and the obligations that participants in 
accident investigations conducted by another State have under International Civil Aviation 
Organization (ICAO) Annex 13 not to divulge certain information. 

Some of the JATR team members’ recommendations for the FAA are specifically related to 
Boeing and/or the B737 MAX, and these recommendations are clearly stated as such. Because 
the JATR team was also chartered to identify potential enhancements to the certification process, 
many of the team members’ recommendations are general in nature. These general 
recommendations should not necessarily be construed as a reflection on Boeing or the B737 
MAX specifically, but rather as opportunities identified by the JATR team members for the FAA 
to improve the certification process. 

The FAA, European Aviation Safety Agency (EASA), and Transport Canada Civil Aviation 
previously conducted B737 MAX development assurance certification/validation reviews. The 
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JATR team’s findings below are not intended to invalidate the results of those reviews but rather 
to supplement them with a more focused, event-driven review. 

The order of the observations, findings, and recommendations does not imply any level of 
importance or priority. 

The Certification Process 

1. Changed Product Rule 


Recommendation R1 

Based on the JATR team’s observations and findings related to the application of the 
Changed Product Rule to the certification of the flight control system of the B737 MAX, 
JATR team members recommend that the FAA work with other civil aviation authorities to 
revise the harmonized approach to the certification of changed products. Changed Product 
Rules (e.g., 14 CFR §§ 21.19 & 21.101) and associated guidance (e.g., Advisory Circular 
21.101-1B and FAA Orders 8110.4C and 8110.48A) should be revised to require a top-down 
approach whereby every change is evaluated from an integrated whole aircraft system 
perspective. These revisions should include criteria for determining when core attributes of 
an existing transport category aircraft design make it incapable of supporting the safety 
advancements introduced by the latest regulations and should drive a design change or a 
need for a new type certificate. The aircraft system includes the aircraft itself with all its 
subsystems, the flight crew, and the maintenance crew. 

These Changed Product Rule revisions should take into consideration the following key 
principles: 

• A comprehensive integrated system-level analysis recognizing that in this complex 
interactive system, every change could interact with other parts of the system. 

• The assessment of proposed design changes on existing systems at the aircraft level 
includes using development assurance principles, system safety principles, and 
validation & verification techniques. The level of assessment should be proportional 
to the impact of the change at the aircraft level. 

• The consideration of training and qualification offlight and maintenance personnel, 
as well as detailed explicit procedures for the safe operation of the aircraft. 


Recommendation R1 is based on the following observations, findings, and supporting 
recommendations related to the JATR team’s review of how the Changed Product Rule process 
was applied to the certification of the B737 MAX. In achieving Rl, JATR team members advise 
the FAA to take actions that include, but are not necessarily limited to, the supporting 
recommendations below. 
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• Recommendation R1.1: The FAA, in collaboration with other CAAs, should: 

(a) Revise the hannonized approach to certifying changed products to achieve the 
expectations of a top-down approach intended by 14 CFR 21.101, where every 
change is evaluated from an integrated, whole aircraft/human system engineering 
perspective and where the whole aircraft is assumed affected by the change(s) until 
substantiated otherwise. This approach should focus on a safe design that as a by¬ 
product leads to compliance with regulatory requirements. 

(b) Develop criteria for determining when core attributes of an existing design make it 
incapable of supporting the safety advancements introduced by the latest regulations 
and therefore warrant consideration of a design change and/or certification under a 
new type certificate. 

(c) Expand the guidance as to what constitutes a substantial change and what can be 
considered as only a significant change to address such aspects as changes in 
software, changes in the roles and responsibilities of the flight crew, and changes to 
maintenance practices. 

o Finding Fl.l-A: Although many aspects of the B737 design have been required to 
meet updated certification requirements each time the type certificate has been 
amended over the years, some elements of the design and certification remain 
rooted in the original 1967 certification of the B737-100. The basic federated 
architecture of the B737 has remained largely unchanged from its original 
conception and drove many design decisions more than 50 years after it was 
originally designed. 10 In the intervening 50 years, significant advancements have 
been made in design methodologies and tools and in analysis and analytical tools, 
which have led to significant improvements in the safety of air transportation. 
While some of these advancements and associated design concepts have been 
incorporated into the B737 MAX, others have been determined to be impractical 
for incorporation into the B737 MAX design or certification requirements using 
current regulations and policies. 

o Observation 01.1 - A: There are no criteria for detennining when the core 

attributes of an existing design make it fundamentally incapable of supporting the 
safety advancements introduced by the latest amendments to airworthiness 
standards. 


10 “Federated architecture” refers to a style of avionics architecture in which each digital flight control function (e.g., 
autopilot, autothrottle, flight management) has its own fault-tolerant computer system, which is only loosely coupled 
to the computer systems of other functions. 
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o Finding Fl.l-B: The guidance given in AC 21.101-IB, Establishing the 

Certification Basis of Changed Aeronautical Products, as to what constitutes a 
substantial change that requires a new type certificate is insufficient. This 
guidance is focused on large-scale structural changes and does not consider 
changes in software or changes in the roles of either the flight crew or the 
maintenance crew to be “substantial changes.” 

o Observation 01.1 -B: Section 21.101 was designed to lead to a top-down approach 
to the certification of changed products whereby the whole aircraft is assumed to 
be affected by changes, and compliance with current amendments must be found 
at the level of the aircraft, unless otherwise noted. 

o Finding Fl.l-C: Boeing’s compliance submissions to the FAA followed a bottom- 
up approach whereby each change and areas affected by a change were presented 
separately, showing compliance at the level of the specific regulation and its 
application to a given change. 

o Finding FI. 1-D: Several risk and failure analyses were mostly done at the level of 
the change and subsystem and not at an integrated aircraft level. 

o Finding FI. 1-E: Some of the Boeing engineers the JATR team spoke with 

described the Boeing process in a manner that reflected an emphasis on meeting 
individual certification requirements, without necessarily having an appreciation 
for the overall safety-based reasons for those requirements. 

• Recommendation R1.2: The FAA, in collaboration with other CAAs, should expand the 
certification process to include “change, areas affected by the change, and areas affecting 
a change .” This expansion should allow for the identification of interactions such as the 
one between the angle of attack (AO A) system and MCAS in the case of the B737 MAX. 

o Observation 01.2-A: The certification process of a derivative aircraft is focused 
on “change and areas affected by the change.” The AOA sensing system of the 
B737 MAX was not changed from the B737 NG and was not affected by any of 
the changes, including the MCAS. 

o Observation 01.2-B: Based on preliminary accident infonnation, both B737-8 
MAX accidents appear to have involved an interaction between the AOA system 
and MCAS. 

• Recommendation R1.3: The FAA should implement mandatory aircraft-level reviews 
along the certification process. These reviews should require risk and failure analyses at 
the integrated aircraft system-level including the flight crew. 
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o Finding F1.3-A: The certification process is focused on a large number of small 
details which may minimize the opportunity for a “big picture” view. 

• Recommendation R1.4: The FAA should provide clear definitions of key terms in its 
guidance for 14 CFR §§ 21.19 and 21.101. 

o Finding F1.4-A: The FAA and Boeing adhered to the applicable regulations, 
policy, and guidance that existed at the time of application for determining 
whether a new type certificate (versus amended type certificate) would be 
required, and for detennining the certification basis for the B737 MAX. 

o Finding FI .4-B: During the certification process of the B737 MAX, FAA 
personnel had no doubt about its suitability to qualify as a derivative aircraft 
which does not require a new type certificate. 

o Finding F1.4-C: The existing rules and guidance governing the certification 

process leave much room for interpretation. Key tenns such as “substantial” as in 
“substantially complete investigation” in § 21.19, as well as “affect” as in “areas 
affected by the change” and “material” as in “contribute materially to the level of 
safety” in § 21.101, are not clearly defined. 

• Recommendation R1.5: The FAA should define and clearly describe the intent and 
expected use of an ADRC in available guidance. In addition, the FAA should elaborate 
on the application of ADRCs in future developments (e.g., future applicant modification 
and supplemental type certificates (STCs)). The FAA should identify the legal standing 
that ensures the adherence to ADRCs for future changes. 

o Observation 01.5-A: ADRCs are identified in the certification basis for the B737 
MAX, but the term is not defined in FAA directives or guidance. The tenn also 
appears in other aircraft type certificate data sheets (TCDSs). 

• Recommendation R1.6: The FAA should develop processes for identifying perceptions of 
vagueness and ambiguity in its guidance and strive to clarify all certification guidance 
that is deemed vague or incomplete. 

o Observation 01.6-A: Different FAA officials the JATR team spoke with 
demonstrated differences in opinions about how to apply the Changed Product 
Rule. 

o Observation 01.6-B: Some FAA officials the JATR team spoke with complained 
about vague and partial guidance for the implementation of the Changed Product 
Rule. 
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• Recommendation R1.7: The FAA and applicants should develop, validate, and 
implement analytical tools appropriate for the analysis of complex systems. 

o Finding F1.7-A: The requirements of an amended type certificate certification 
process to focus only on “change and areas affected by the change” may fail to 
recognize that the whole aircraft system (including the flight crew) could be 
affected by seemingly small changes. 

o Observation 01.7-A: A complicated system is characterized by a linear 

relationship between cause and effect, whereas a complex system is characterized 
by a non-linear relationship between cause and effect, such that small causes 
could lead to very large effects. 

o Finding FI ,7-B: Although the aircraft itself may only be a complicated system, 
the aircraft system including the flight crew is a complex system. 

o Finding F1.7-C: Analytical tools designed for complicated systems may not be 
sufficient for the analysis of complex systems. 

• Recommendation R1.8: The FAA should ensure that the TCDS for the B737 MAX 
(TCDS No. A16WE) clearly states which part of 14 CFR 25.1322 (Flightcrew Alerting), 
and at which amendment level, the B737 MAX complies to. 

o F1.8-A: The A16WE TCDS does not clearly state the applicable amendment level 
that the B737 MAX complies with for § 25.1322. Significant review and 
background knowledge is required to determine that the B737 MAX is compliant 
with the following: 

■ Arndt 131: (a), (b)(1), (c)(1), (e), and (f). 

■ Arndt 38: none, even though this is the certification basis of the B737 NG. 

■ Arndt NA: none, § 25.1322 not present in initial part 25 regulation. 

■ ADRC: (b)(2), which is technically equivalent to Arndt 38 para (b), and 
(c)(3). 

• Recommendation R1.9: The FAA should ensure that TCDSs accurately reflect when 
compliance is found at the stated amendment level and when compliance is limited to a 
subset of the aircraft (such as a change). 

o Finding F1,9-A: Compliance with regulatory requirements and the certification 
process as a whole are the results of a negotiation process between the applicant 
and the FAA. The TCDS contains the certification basis for the aircraft, which is 
the output of the Changed Product Rule process. In some cases, for example 
where compliance is limited to a change rather than to the whole aircraft, this 
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negotiated agreement between the applicant and the FAA is not consistently 
documented. For example, the certification basis for the B737 MAX documented 
exceptions to the application of some amendments, but not others. Exceptions are 
listed for 14 CFR 25.607 (typical of about 40 regulations that had exceptions), 
and not for 14 CFR 25.1302. This indicates that § 25.1302 applies to the entire 
aircraft; yet, § 25.1302 was only partially applied on the B737 MAX. 

o Finding F1.9-B: Because the aircraft alerting system is not designed to comply 
with all aspects of the latest amendment of § 25.1322 (Amendment 131), 

§ 25.1302 cannot be fully applicable. These rules depend on each other, and this 
is further indication that § 25.1302 was not applied to the entire aircraft despite 
the TCDS indicating that it was. 

2. Development and use of up-to-date requirements and practices 


Recommendation R2 

Based on the JATR team’s observations and findings related to the regulations, policy, and 
compliance methods applied to the B737MAX, JATR team members recommend that the 
FAA update regulations and guidance that are out of date and update certification 
procedures to ensure that the applied requirements, issue papers, means of compliance, and 
policies fully address the safety issues related to state-of-the-art designs employed on new 
projects. JATR team members also recommend that the FAA review its processes to ensure 
that regulations and guidance materials are kept up to date. 


Recommendation R2 is based on the following observations, findings, and supporting 
recommendations related to the JATR team’s review of the regulations, policy, and compliance 
methods applied to the certification of the B737 MAX. In achieving R2, JATR team members 
advise the FAA to take actions that include, but are not necessarily limited to, the supporting 
recommendations below. 

• Recommendation R2.1: The FAA should review the scope of 14 CFR 25.1302 (Installed 
Systems and Equipment for Use by the Flightcrew) applicability and clearly define in the 
TCDS the approach taken for certification. 

o Observation 02.1-A: MCAS, although a significant functional change, was never 
highlighted as an area requiring additional scrutiny from a human factors 
perspective. As a result, no human factors test cases were designed to investigate 
the adequacy of the design. 


11 




Joint Authorities Technical Review 
Boeing 737 MAX Flight Control System 
Observations, Findings, and Recommendations 


• Recommendation R2.2: The FAA should update AC 25.1302-1, Installed Systems and 
Equipment for Use by the Flightcrew, to clarify the acceptability (or not) of using 

14 CFR 25.1302 in changed areas. 

o Observation 02.2-A: The application of § 25.1302 to areas of change is not 

explicitly described in its associated guidance material, AC 25.1302-1. The intent 
of § 25.1302 is stated as follows in its introductory paragraph: 

This section applies to installed systems and equipment intended 
for flightcrew members’ use in operating the airplane from their 
normally seated positions on the flight deck. The applicant must 
show that these systems and installed equipment, individually 
and in combination with other such systems and equipment, are 
designed so that qualified flightcrew members trained in their 
use can safely perform all of the tasks associated with the 
systems' and equipment's intended functions. 

o Finding F2.2-A: The JATR team’s assessment is that the design and evaluation 
aspects should be considered for the whole of the cockpit environment, and not to 
components in isolation. 

• Recommendation R2.3: The FAA should expedite a rule change to 14 CFR 25.1309 
(Equipment, Systems, and Installations) and its associated means of compliance in order 
to implement the recommendations stemming from the Aviation Rulemaking Advisory 
Committee (ARAC) Systems Design and Analysis Hannonization Working Group 
(SDAHWG) (2001). This action is necessary to minimize the possibility of applicants 
using old guidance that is not fully effective for the system development and for 
conducting SSA in the context of increased system complexity and interactions. 

o Finding F2.3-A: Although the certification basis for § 25.1309 was updated for 
the latest amendment per Changed Product Rule analysis, delayed FAA 
rulemaking for updating § 25.1309 and related guidance according to the 
recommendations of the ARAC SDAHWG allows applicants to use geriatric 
guidance for safety assessment demonstration. 

• Recommendation R2.4: The FAA should evaluate applicants’ procedures for detennining 
the need of any subsystem and any change to show compliance with a regulatory 
requirement. Special attention should be paid to compliance with new requirements. 

o Observation 02.4-A: Boeing did not identify MCAS as requiring compliance with 
§ 25.1302, and MCAS was not assessed for such compliance. 
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• Recommendation R2.5: Sufficient time and resources should be allocated for the proper 
treatment of issue papers to avoid inconsistencies and errors. 

o Observation 02.5-A: Issue papers document the negotiation process between the 
applicant and the FAA to detennine the certification basis of a product, establish 
means of compliance, and resolve other issues. 

o Observation 02.5-B: All issue papers must be closed prior to granting 
certification. 

o Finding F2.5-A: Some closed B737-8 MAX issue papers contain inconsistencies. 
For instance, in Issue Paper 0-1, MDR and ODR stand for different definitions 
between Boeing’s position and the FAA’s response. 

o Finding F2.5-B: Some B737-8 MAX issue papers contain typographical and 
grammatical errors which could indicate a hurried process. 

• Recommendation R2.6: The FAA should review its internal procedures to emphasize the 
need for issue papers when the applicant proposes means of compliance that deviates 
from advisory circulars. 

o Observation 02.6-A: A combination of ACs was used for demonstrating 
compliance with system safety requirements; no AC/acceptable means of 
compliance (AMC) was followed in its entirety. The detailed use of the 
referenced ACs and an indication of which sections are applicable was not 
formally recorded in any certification document that the JATR team reviewed. 

o Finding F2.6-A: The use of a combination of partial ACs as means of compliance 
should have led the FAA to formalize the agreement with this strategy, possibly 
by means of an AMC issue paper. 

• Recommendation R2.7: If any flight control surface is used in a novel manner, the FAA 
should be directly involved. The FAA should assess the need for an issue paper for 
development of acceptable means of compliance with existing regulations, or develop 
special conditions if the regulations do not contain adequate or appropriate safety 
standards. 

o Finding F2.7-A: The FAA was not completely unaware of MCAS; however, 
because the information and discussions about MCAS were so fragmented and 
were delivered to disconnected groups within the process, it was difficult to 
recognize the impacts and implications of this system. If the FAA technical staff 
had been fully aware of the details of the MCAS function, the JATR team 
believes the agency likely would have required an issue paper for using the 
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stabilizer in a way that it had not previously been used. MCAS used the stabilizer 
to change the column force feel, not trim the aircraft. This is a case of using the 
control surface in a new way that the regulations never accounted for and should 
have required an issue paper for further analysis by the FAA. If an issue paper had 
been required, the JATR team believes it likely would have identified the 
potential for the stabilizer to overpower the elevator. 

• Recommendation R2.8: The FAA should establish appropriate pilot recognition times 
and reaction times, based on substantive scientific studies which take into account the 
operational environment, the circumstances under which malfunctions may occur, and the 
effect of surprise. 

o Observation 02.8-A: FAA guidance for test flights in AC 25-7D, Flight Test 
Guide for Certification of Transport Category Airplanes, and AC 25.1329-1C, 
Approval of Flight Guidance Systems, require test pilots to delay initiation of 
response to flight control or flight guidance malfunctions to account for pilot 
recognition time and pilot reaction time. Often, recognition time is assumed to be 
1 second, and reaction time is assumed to be 3 seconds. Thus, test pilots are told 
that “Recovery action should not be initiated until 3 seconds after the recognition 
point” (AC 25.1329-1C). 

o Observation 02.8-B: The current guidance recognizes that pilot recognition time 
may depend on various factors including the nature of the failure, but applicants 
are only required to prepare specific justification of their assumed recognition 
time if it is less than 1 second. 

o Observation 02.8-C: Although the above guidance is aimed at test pilots 
conducting test flights, applicants seem to use this guidance as a design 
assumption that the pilot will be able to respond correctly within 4 seconds of the 
occurrence of a malfunction. For example, in the case of the B737 MAX, it was 
assumed that, since MCAS activation rate is 0.27 degrees of horizontal stabilizer 
movement per second, during the 4 seconds that it would take a pilot to respond to 
an erroneous activation, the stabilizer will only move a little over 1 degree, which 
should not create a problem for the pilot to overcome. 

o Observation 02.8-D: No studies were found that substantiate the FAA guidance 
concerning pilot recognition time and pilot reaction time. 

o Observation 02.8-E: Several FAA studies with general aviation pilots 

demonstrate that these general aviation pilots may take many seconds, and in 
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some cases many minutes, to recognize and respond to malfunctions (e.g., 
DOT/FAA/AM-97/24; DOT/FAA/AM-02/19; DOT/FAA/AM-05/23). 

o Observation 02.8-F: A NASA study of abnonnal flight events with qualified, 
current, and active airline pilots also found substantially longer recognition times 
and reactions times, even in the case of expected events, than the times given in 
AC 25-7D and AC 25.1329-1C. 11 

o Observation 02.8-G: Analysis of aviation accidents demonstrates that pilots may 
take a significantly longer time to recognize a malfunction and respond to it than 
the test flight guidance suggests. For example, the NTSB states: “When a flight 
crew is confronted with a sudden, abnonnal event, responses are more likely to be 
delayed or inappropriate.” (NTSB/AAR-14/01) 

o Observation 02.8-H: Modem aircraft can have subtle failure modes that may take 
substantial amounts of time to be recognized. Furthennore, automation can mask 
some failures and significantly delay the possibility for the pilot to recognize the 
malfunction. 

o Finding F2.8-A: It is not clear on what the FAA guidance concerning pilot 
recognition time and pilot reaction time is based. 

o Finding F2.8-B: Pilot recognition time and reaction time to a malfunction may 
depend on the particular nature of the malfunction, the circumstances under which 
it occurs, the corrective action required, and the individual pilot. 

o Finding F2.8-C: There is a substantial difference between the situation of a test 
pilot who is testing a particular malfunction with precise foreknowledge of the 
malfunction to be tested and the proper response to be initiated, and the situation 
of a line pilot on a routine revenue flight who is not expecting any malfunction. 
Thus, guidance that is relevant to test flights may not be appropriate for routine 
revenue flights. 

o Finding F2.8-D: The 3-second reaction time assumption dates back decades, to 
where the performance of the autopilot was constantly monitored by the crew in 
flight (e.g., guidance given in AC 25.1329-1 A, Automatic Pilot Systems Approval, 
dated July 8, 1968). However, with increasing reliability and advances in flight 


11 Casner, S.M., R.W. Geven, and K.T. Williams (2013). The Effectiveness of Airline Pilot Training for Abnormal 
Events, Human Factors, 55, 477-485. 
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deck alerting and displays, it may no longer be appropriate to assume that the 
pilot flying will be monitoring the automation as closely as in the past. 

o Finding F2.8-E: The FAA’s guidance concerning pilot reaction time of 3 seconds 
may not be appropriate given current aircraft technology and the current 
operational environment. 

o Finding F2.8-F: Although current guidance seems to recognize potential 

variability in pilot recognition time, it is not clear that applicants are following the 
spirit of that guidance, because only recognition times of less than 1 second must 
be formally justified. 

• Recommendation R2.9: The FAA should require applicants to provide validated and 
justified pilot recognition and reaction times for any given failure, with consideration of 
all associated flight deck effects within the expected operational environment. 

o This recommendation is based on Observations 02.8-A through 02.8-H and 
Findings F2.8-A through F2.8-F, above. 

• Recommendation R2.10: The FAA should provide guidance to test pilots to initiate 
recovery action only once the combined recognition time and reaction time validated for 
the given failure being tested have elapsed. 

o This recommendation is based on Observations 02.8-A through 02.8-H and 
Findings F2.8-A through F2.8-F, above. 

3. Consistent interpretation and application of requirements 


Recommendation R3 

Based on the JATR team ’s observations and findings related to the certification of the B73 7 
MAXflight control system and related interfaces, JATR team members recommend that the 
FAA review the B 73 7 MAX compliance to 14 CFR §§ 25.1329 (Flight Guidance System), 
25.1581 (Airplane Flight Manual - General), and 25.201 (Stall Demonstration) and ensure 
the consistent application and interpretation of regulatory guidance material for the system 
safety assessment, handling qualities rating method, and conformity requirements for 
engineering simulators and devices. Should there be a non-compliance, the root cause 
should be identified and measures implemented to prevent recurrence. 


Recommendation R3 is based on the following observations, findings, and supporting 
recommendations related to the JATR team’s review of the certification of the B737 MAX flight 
control system and related interfaces. In achieving R3, JATR team members advise the FAA to 
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take actions that include, but are not necessarily limited to, the supporting recommendations 
below. 

• Recommendation R3.1: The FAA should ensure early involvement by applicants and the 
FAA in the establishment of the detailed means of compliance for SSA demonstration 
(e.g., 14 CFR §§ 25.1309 (Equipment, Systems, and Installations) and 25.671 (Control 
Systems - General)), especially in case any deviations from standard guidance are 
planned, or if additional guidance not originally intended for §§ 25.1309 and 25.671 is 
expected to be part of the compliance demonstration. 

o This recommendation is based on Observation 02.6-A and Finding F2.6-A, 
above. 

• Recommendation R3.2: The FAA should issue a policy statement on the need for caution 
and early negotiation with the certification authority when an applicant proposes using 
additional guidance not originally intended for showing compliance to system safety 
requirements. 

o Observation 03.2-A: The JATR team observed that the SSA takes credit for the 
probability that the aircraft will be flying in certain portions of the flight envelope, 
as documented in AC 25-7C. A probability of IE-3 for the aircraft in the 
operational flight envelope (OFE) was used in combination with the probability of 
the system failure to achieve the IE-7 minimum probability required for the 
“hazardous” MCAS failure condition. Use of AC 25-7C is not a standard industry 
approach for § 25.1309 compliance. The JATR team’s view of the intent of the 
probability of IE-3 for the OFE in the HQRM is to select flight test cases for 
handling qualities evaluation, not to support the quantitative aspects of 
§§ 25.1309 or 25.672(c) compliance. 

• Recommendation R3.3: The FAA should implement policy that emphasizes compliance 
with “safe and reliable” guidance (e.g., AC 25-22, Certification of Transport Airplane 
Mechanical Systems ) for establishing minimum reliability requirements for system 
functions used for flight requirements demonstration in addition to the minimum 
reliability safety requirements defined by the FHA process. 

o Observation 03.3-A: The JATR team observed that the minimum required 
probability for the loss of MCAS is IE-3 as recorded in the internal safety 
requirements, i.e., consistent with the “minor” classification in the FHA. “Safe 
and reliable” guidance for system functions used for compliance with 14 CFR part 
25, subpart B requirements was not considered during development, nor required 
by the FAA. Because quantitative analysis for “minor” failure conditions is not 
required, it is unclear if the MCAS design would be considered safe and reliable 
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to be used as an augmentation function for compliance with flight requirements 
under 14 CFR part 25, subpart B. 

• Recommendation R3.4: The FAA should review the natural (bare airframe) stalling 
characteristics of the B737 MAX to determine if unsafe characteristics exist. If unsafe 
characteristics exist, the design of the speed trim system (STS)/MCAS/elevator feel shift 
(EFS) should be reviewed for acceptability. 

o Observation 03.4-A: The original implementation of MCAS was driven primarily 
by its ability to provide the B737 MAX with FAA-compliant flight characteristics 
at high speed. An unaugmented design would have been at risk of not meeting 
14 CFR part 25 maneuvering characteristics requirements due to aerodynamics. 

o Observation 03.4-B: Extension of MCAS to the low-speed and lg environment 
during the flight program was due to unacceptable stall characteristics with STS 
only. The possibility of a pitch-up tendency during approach to stall was 
identified for the flaps-up configuration prior to the implementation of MCAS. 

o Finding F3.4-A: The acceptability of the natural stalling characteristics of the 
aircraft should form the basis for the design and certification of augmentation 
functions such as EFS and STS (including MCAS) that are used in support of 
meeting 14 CFR part 25, subpart B requirements. 

• Recommendation R3.5: The FAA should review 14 CFR 25.201 (Stall Demonstration) 
compliance for the B737 MAX and determine if the flight control augmentation functions 
provided by STS/MCAS/EFS constitute a stall identification system. 

o Finding F3.5-A: The nose-down pitch identified during Boeing flight tests for 
stall appears to the JATR team to be the product of system augmentation with 
flaps and gear up, and is likely due to stabilizer motion from the MCAS function. 

o Finding F3.5-B: The FAA-accepted Boeing flight test technique of freezing 
column deflection at the onset of EFS was perceived by the JATR team as 
possibly not meeting the requirements of § 25.201 for natural stall identification 
from nose-down pitch, not readily arrested. Column/elevator deflection data 
indicates that there may be an insufficient column input to attempt to arrest the 
nose-down pitch created by system augmentation. 

o Finding F3.5-C: The JATR team considers that the STS/MCAS and EFS 
functions could be considered as stall identification systems or stall protection 
systems, depending on the natural (unaugmented) stall characteristics of the 
aircraft. From its data review, the JATR team was unable to completely rule out 
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the possibility that these augmentation systems function as a stall protection 
system. 

• Recommendation R3.6: The FAA should review the use of non-standard flight test 
techniques, such as freezing column position at EFS actuation, when showing compliance 
with 14 CFR 25.201 (Stall Demonstration). The use of non-standard flight test techniques 
may not meet the associated regulatory requirements. 

o This recommendation is based on Findings F3.5-A, F3.5-B, and F3.5-C, above. 

• Recommendation R3.7: The FAA should review how compliance was shown for the stall 
identification system on the B737 MAX with respect to inadvertent operation due to 
single failures. 

o Finding F3.7-A: The JATR team considers that system features on the B737 
MAX might constitute a stall identification system. This system is vulnerable to 
inadvertent actuation due to a single failure, which would not meet the accepted 
guidance contained within AC 25-7C, Chapter 8, Section 228. 

• Recommendation R3.8: The FAA should review the prescriptive use of 3 seconds under 
14 CFR 25.255 (Out-of-Trim Characteristics) for the evaluation of mis-trim conditions, 
especially for automatic trim systems where pilot recognition is relied upon to detect and 
arrest runaway failures. The rate of trim used by these automatic systems should also be 
considered in showing compliance to § 25.255. 

o Observation 03.8-A: Out-of-trim characteristics, per the requirements of 

§ 25.255, were found acceptable for a 0.6 unit nose-down out-of-trim condition. 
This out-of-trim value was determined by 3 seconds of trim input at the flaps-up 
main electric stabilizer trim rate of 0.2 degrees per second, which is greater than 
the autopilot trim rate. 

o Observation 03.8-B: The higher MCAS trim rate of 0.27 degrees per second was 
not selected for the demonstration of compliance with § 25.255, even though 
failures could result in un-commanded stabilizer trim movement at this rate. 

o Finding F3.8-A: Section 25.255 applies to jet upset events and uses a prescriptive 
3 seconds as the amount of out-of-trim that could occur before pilot reaction. For 
automatic trim systems, the 3-second reaction time may not be appropriate, 
depending on the cockpit alerting philosophy and trim system architecture and 
controls. 

• Recommendation R3.9: The FAA should review the AFM procedure for stabilizer 
runaway and ensure that adequate emphasis is placed on the importance of using main 
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electric stabilizer trim to return to a trimmed state. Crew error should be considered in the 
event that aisle stand stabilizer cutout switches are used before returning to trim 
conditions. 

o Finding F3.9-A: Certain stabilizer runaway failures may generate significant out- 
of-trim conditions. Main electric stabilizer trim is considered the primary means 
to stop runaway stabilizer in Boeing’s assumptions and validation tests. The 
degree of stabilizer mis-trim and resulting transient from steady-state flight may 
result in hazardous or even catastrophic failure conditions. 

• Recommendation R3.10: The FAA should review the Boeing assumption of a 4-second 
pilot reaction time to stabilizer runaway failures to ensure that a conservative value is 
used, since pilot action is required to counter these failures. 

o Finding F3.10-A: Manual stabilizer trim wheel forces increase with increased 
speed and degree of out-of-trim condition. The degree of out-of-trim condition is 
dependent on pilot recognition and reaction technique and time. Manual stabilizer 
trim wheel forces could become significant when assumed pilot reaction times are 
reasonably exceeded, especially for high-speed conditions. During stabilizer 
runaway conditions where main electric stabilizer trim is not available, either due 
to system failures or the erroneous selection of stabilizer cutout switches prior to 
returning to trim, the crew must use the manual stabilizer trim wheel to return to a 
trimmed condition. 

• Recommendation R3.11: For failure of the STS, the FAA should consider the 
requirement to alert flight crews to the reduction in safety margins due to the absence of 
the stability augmentation function provided by the system. Consideration should be 
given to AFM flight envelope limitations or warning/caution statements, if required. 

o Observation 03.11-A: STS inoperative wind-up turns were completed to 1.6g as 
part of the B737 MAX certification. STS inoperative stalls were completed to 
stick shaker + 1 second (approach to stall). The JATR team’s assessment is that 
the limited envelope for evaluation of characteristics for this failure condition 
does not support the absence of an envelope limitation in the associated non¬ 
normal procedure. 

o Observation 03.11-B: STS inoperative wind-up turns, flown by Boeing during the 
course of the JATR, did not show any unsafe characteristics to approximately 2g. 

o Finding F3.11-A: HQRM guidance from AC 25-7C was applied for the evaluation 
of control systems malfunctions. The application of the probabilistic aspects of 
this guidance was appropriate to the detennination of the required handling 
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qualities, but may not be suitable for evaluation of the failure condition per AC 
25.1309-1A, System Design and Analysis, and AC 25-7C. 

o Finding F3.11-B: For § 25.1309 compliance, the criticality of the failure condition 
should account for intensifying conditions, such as crew workload or multiple 
cockpit indications, and effects and interrelationship of failures with the flight 
envelopes. 

o Finding F3.11-C: Boeing’s application of HQRM allowed for a reduced envelope 
in the evaluation of SPEED TRIM FAIL, which may not meet the intent of 
guidance within AC 25-7C and AC 25-1309-1A. 

• Recommendation R3.12: Because the guidance provided by the HQRM in AC 25-7D is 
not harmonized, the FAA should determine if continued application of HQRM is 
appropriate for the evaluation of failure conditions and revise the AC accordingly. 

o This recommendation is based on Observations 03.11-A and 03.11-B and 
Findings F3.11 - A, F3.11 -B, and F3.11 -C, above. 

• Recommendation R3.13: The FAA should ensure that simulation devices that are used 
for certification credit have the required level of fidelity for the associated test. 

o Observation 03.13-A: During evaluation in the Boeing engineering simulator (E- 
Cab), the JATR team observed that the device does not incorporate control 
loading on the manual stabilizer trim wheel. As a result, control forces on the 
manual stabilizer trim wheel are not representative of the aircraft. 

• Recommendation R3.14: The FAA should review the B737 MAX’s compliance to 
14 CFR 25.1581 (Airplane Flight Manual - General) and address the inconsistency 
between AC 25.1581-1 and 14 CFR §§ 25.1581 thru 25.1587, which outline the required 
information to be included in the AFM and approved under § 25.1581. 

o Finding F3.14-A: Part 25 regulations require that the AFM be approved by the 
FAA and contain information necessary to safely operate the aircraft, including all 
the normal, non-nonnal, and emergency operating procedures. Contradictory 
guidance in AC 25.1581-1 has allowed Boeing AFMs to minimize the content of 
operating procedures that are subject to FAA scrutiny and approval. The result is 
that the FCOM/Quick Reference Handbook includes most of the operating 
procedures, in accordance with FAA guidance (AC 25.1581-1), and has become 
the master document for procedures to ensure safe operation. The FCOM, which 
is not approved by the FAA, includes systems infonnation that is not included in 
the AFM. Subsequent changes to these procedures can therefore occur without 
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certification oversight. As a result, there is a question about whether compliance 
to § 25.1581 in accordance with AC 25.1581-1 for the B737 MAX meets the 
intent of § 25.1581. 

• Recommendation R3.15: The FAA should exercise careful oversight and scrutiny of 
AFM procedures for Boeing aircraft. 

o This recommendation is based on Finding F3.14-A, above. 

• Recommendation R3.16: The FAA should review the certification of the B737 MAX to 
Amendment 119 for 14 CFR 25.1329 (Flight Guidance System). If necessary, system 
changes should be introduced to ensure compliance and safe operations. 

o Observation 03.16-A: The B737 MAX TCDS shows that the B737 MAX 

complies with Amendment 119 for § 25.1329 at the whole aircraft level with no 
exceptions. 

o Observation 03.16-B: The B737 MAX autopilot does not automatically 

disconnect upon stick shaker activation. The JATR team was unable to determine 
how compliance was shown to Amendment 119 for § 25.1329. 

• Recommendation R3.17: The FAA should review the compliance details of the optional 
head-up display (HUD) approved under STC on the B737 MAX and detennine if its 
alerting meets regulatory requirements. 

o Finding F3.17-A: The JATR team was unable to determine that the third-party 
HUD installed at the factory follows FAA guidance in AC 25-1 IB, Electronic 
Flight Displays, because the team could not conclusively detennine whether the 
HUD includes the IAS DISAGREE, ALT DISAGREE, and AOA DISAGREE 
alerts. Further, the HUD displays an AOA gauge but the head-down display does 
not display an AOA gauge unless the customer requested this option to be 
enabled. The alerting presented to the pilot who is using the HUD will be 
different from the alerting presented to the pilot who is using the head-down 
display. 

Additional Observations 

The JATR team makes the following additional observations: 

• Observation 03.18-A: The high-speed MCAS function was reviewed, and for normal 
operation (not considering failure cases) no concerns were noted. 
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• Observation 03.18-B: Within the limited scope of the E-Cab session conducted by the 
JATR team, no unsafe conditions were noted with MCAS inoperative for high-speed 
wind-up turns. 

• Observation 03.18-C: The certification of the fly-by-wire spoiler system, from a 14 CFR 
part 25, subpart B perspective, appeared to meet all related requirements. 

• Observation 03.18-D: The certification of the B737 MAX for flight in icing was 
reviewed and judged acceptable with respect to 14 CFR part 25, subpart B requirements. 

4. Changes during the certification process 


Recommendation R4 

Based on the JATR team’s observations and findings related to the FAA type certification 
process, JATR team members recommend that the FAA review and update the regulatory 
guidance pertaining to the type certification process with particular emphasis on early FAA 
involvement to ensure the FAA is aware of all design assumptions, the aircraft design, and 
all changes to the design in cases where a changed product process is used. The FAA 
should consider adding feedback paths in the process to ensure that compliance, system 
safety, and flight deck/human factors aspects are considered for the aircraft design 
throughout its development and certification. 


Recommendation R4 is based on the following observations, findings, and supporting 
recommendations related to the JATR team’s review of the FAA type certification process. In 
achieving R4, JATR team members advise the FAA to take actions that include, but are not 
necessarily limited to, the supporting recommendations below. 

• Recommendation R4.1: The FAA should consider defining objective criteria for FAA 
familiarization with design details and FAA involvement in compliance findings, to be 
applied initially and all along the certification process, when development and 
certification prompt design or compliance method revision. 

o Finding F4.1-A: The JATR team finds it unclear how items to be presented to the 
FAA BASOO staff are selected. Other authorities have published guidelines in the 
matter, which typically require items to be presented to the authority based upon 
critically, novelty, or past experience. [Note: In the context of the B737 MAX, the 
JATR team’s assessment is that MCAS should have been considered a novelty 
(and therefore clearly highlighted to the FAA technical staff) owing to the 
important differences in function and implementation it has on the B737 MAX 
compared with the previous MCAS installed on the B767-C2 (tanker).] 
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o Finding F4.1-B: Although some FAA personnel may have been briefed on the 
MCAS function, the JATR team did not have access to the contents of such 
briefings to evaluate the level of infonnation provided to the FAA. In addition, 
based on its review, the JATR team concluded that the content of certification 
deliverables would not have provided FAA technical staff with awareness of key 
details of the MCAS function on the B737 MAX, including architecture, signal 
inputs, and limits of authority. 

o Finding F4.1-C: The JATR team found that the certification plans and some 
certification deliverables (e.g., the preliminary system safety assessment (PSSA)) 
were not updated to describe the expansion of the MCAS function for the low 
Mach portion of the flight envelope and for compliance with stall-related 
requirements. 

• Recommendation R4.2: The FAA should consider developing policy or standards to be 
followed by applicants on proper visibility, clarity, and consistency of key design and 
compliance information that is submitted for certification, particularly with new design 
features. 

o Finding F4.2-A: As an amended type certificate under the Changed Product Rule 
(§ 21.101), many B737 MAX certification deliverables consisted of revisions to 
B737 NG certification documents. As a result, the MCAS description, including 
architecture, interfaces, logics, etc., is fragmented among several documents. 

o Finding F4.2-B: Although MCAS may have been briefed to some FAA personnel, 
key aspects of the MCAS function such as intended function description, its 
interfaces, and architecture, were not directly visible to the FAA in a 
straightforward manner through the certification deliverable documents. 

• Recommendation R4.3: The FAA should implement policy or further guidance that 
emphasizes the need for early coordination with the certification authority for the FHA 
validation and PSSA review to ensure the proposed system architecture can reasonably 
meet the FHA safety requirements. In addition, the FAA should emphasize that early 
involvement with the certification authority is recommended for design changes. 

o Finding F4.3-A: The FAA certification process resulted in FHA/ PSSA 

infonnation being submitted much too late (at type inspection authorization) for 
the FAA to have any influence on the proposed MCAS design for the purpose of 
demonstrating compliance. The FHA information that is delivered to the FAA is 
the FHA summary. Therefore, the FAA does not have the details of the analysis, 
which are documented in Boeing’s internal coordination sheets (including 
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important FHA assumptions). FAA’s visibility into important system safety 
information was therefore incomplete and fragmented. 

• Recommendation R4.4: The FAA should refuse to accept function descriptions that are 
fragmented among several documents. 

o This recommendation is based on Finding F4.2-A, above. 

• Recommendation R4.5: The FAA should require applicants to highlight and properly 
describe any functional change at the earliest stage possible in the certification process 
regardless of the preliminary functional hazard classification. 

o This recommendation is based on Findings F4.1-A, F4.1-B, and F4.1-C, above. 

• Recommendation R4.6: The FAA should ensure applicants maintain records of 
interactions with certification authorities, especially if those interactions lead to 
agreements affecting documentation and certification deliverables. 

o Observation 04.6-A: Regarding aircraft-level safety analyses, the aircraft 
functional hazard assessment was included as a certification deliverable to the 
FAA in the “Airplane Level FHA and Development Assurance process 
Certification Plan” (13449 Revision F), but the aircraft safety assessment was not. 
The JATR team was informed that there was an agreement with the certification 
authorities not to include the aircraft safety assessment in the certification plan, 
but the team could not find the recording of such agreement outside of the 
agreement inherent in the acceptance of the certification plan. 
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5. Delegation of certification authority 


Recommendation R5 

Based on the JATR team’s observations and findings related to FAA ’s oversight by the Boeing 
Aviation Safety Oversight Office (BASOO), JATR team members recommend that the FAA 
conduct a workforce review of the BASOO engineer staffing level to ensure there is a 
sufficient number of experienced specialists to adequately perform certification and oversight 
duties, commensurate with the extent of work being performed by Boeing. The workforce 
levels should be such that decisions to retain responsibility for finding compliance are not 
constrained by a lack of experienced engineers. 

The FAA should review the Boeing Organization Designation Authorization (ODA) work 
environment and ODA manual to ensure the Boeing ODA engineering unit members (E-UMs) 
are working without any undue pressure when they are making decisions on behalf of the 
FAA. This review should include ensuring the E-UMs have open lines of communication to 
FAA certification engineers without fear ofpunitive action or process violation. 


Recommendation R5 is based on the following observations, findings, and supporting 
recommendations related to the JATR team’s review of the FAA’s oversight of the Boeing ODA. 
In achieving R5, JATR team members advise the FAA to take actions that include, but are not 
necessarily limited to, the supporting recommendations below. 

• Recommendation R5.1: The FAA should identify and implement procedures for 

increased direct FAA involvement in safety critical areas of ODA certification projects. 
Safety critical areas may include certain regulations, reports, inspections, tests, or other 
critical items. Direct involvement may include the FAA retaining approvals, conducting 
real-time oversight, or implementing other procedures. 

o Observation 05.1-A: The FAA initially delegated acceptance of approximately 
40% of the B737 MAX project’s certification plans to the Boeing ODA. 
Additional certification plans that were originally retained for acceptance by the 
FAA were later delegated to the Boeing ODA as the certification project 
progressed. While the JATR team did not conduct an exhaustive review of other 
ODAs, the team observed that delegating the acceptance of certification plans 
does not appear to be a widespread practice for the FAA. 

o Finding F5.1-A: The FAA extensively delegated compliance findings on the 
B737-8 MAX project to the Boeing ODA. Safety critical areas, including system 
safety documents related to MCAS, were initially retained by the FAA and then 
delegated to the Boeing ODA. (See also Findings F4.1-A, F4.1-B, and F4.1-C.) 
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o Finding F5.1-B: The JATR team’s belief is that FAA involvement in the 

certification of MCAS would likely have resulted in design changes that would 
have improved safety. 

• Recommendation R5.2: The FAA should conduct a workforce review of the BASOO 
engineer staffing level to ensure sufficient personnel to adequately perform all assigned 
duties (including but not limited to: certification document approval, findings of 
compliance, and ODA oversight). 

o Observation 05.2-A: The certification process of the B737-8 MAX was extensive 
and produced a large number of large documents. 

o Observation 05.2-B: The Boeing ODA organization is staffed by approximately 
1,500 people, whereas the FAA’s BASOO is staffed by 45 people. 

o Finding F5.2-A: There may be a lack of capacity and depth of experience of 
BASOO engineering members to approve and make findings of compliance for 
retained items. 

a) Out of 45 BASOO personnel, there are 18 working-level engineers and 6 
senior engineers (24 engineers total). The JATR team was unable to 
conclusively detennine the levels of experience of the working-level 
engineers and understands at least some of them may be entry-level engineers. 
Depending on the number of entry-level engineers in the BASOO, there could 
be an imbalance of working-level engineers in relation to the number of senior 
engineers reasonably expected to be required given the complexity of work by 
Boeing. 

b) Depending on the number of working-level engineers who are entry-level, 
there could be a training burden that may further impact the capacity of senior 
engineers and program managers. 

o Finding F5.2-B: The allocated staffing levels of 24 BASOO engineers may not be 
sufficient to carry out the work associated with retained items and with the 
conduct of oversight duties. 

a) BASOO engineers are required to review and find compliance for retained 
items as well as conduct on-site and desk audits of the Boeing ODA. 

b) The BASOO may not be sufficiently staffed to review all the Boeing 
programs (737, 747, 767, 777, and 787). There are two technical staff 
assigned per Boeing program. Some of the technical staff are new engineers 
with limited airworthiness experience. 
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o Finding F5.2-C: BASOO engineers may not have had the technical insight, due 
to lack of involvement, to assess compliance. 

• Recommendation R5.3: The FAA should review the Boeing work environment for E- 
UMs to ensure the FAA requirements for undue pressure are being complied with such 
that E-UMs have an acceptable environment to perfonn certification work on behalf of 
the FAA. 

o Observation 05.3-A: FAA Order 8100.15B, Organization Designation Authorization 
Procedures, paragraph 3-6, calls on ODAs to administer duties for the FAA without 
undue pressure or influences from other organizational segments or individuals. 

o Finding F5.3-A: There are signs of undue pressure on E-UMs perfonning delegated 
functions, which may be attributed to conflicting priorities and an environment that 
does not support FAA requirements. 

o Observation 05.3-B: The BASOO conducted oversight interviews of the E-UMs, 
resulting in a finding and associated corrective action pertaining to undue pressure. 

• Recommendation R5.4: The FAA should review ODA procedures in order to remove 
undue burdens and barriers between the Boeing ODA and the FAA and promote cultural 
changes at both organizations. 

o Observation 05.4-A: The FAA is responsible for establishing the certification 
basis for new and changed products, including guidance to be followed. Boeing, 
as the applicant, was ultimately responsible for the design, compliance with the 
defined certification basis, and delivery of a safe product. The FAA is not able to 
properly perfonn its reviews if the technical staff is missing an adequate level of 
information on the proposed change. 

o Observation 05.4-B: In its review of Boeing’s ODA manual and related Boeing 
Process Instructions, the JATR team observed that Boeing’s procedures could be 
improved to facilitate a culture where Boeing and the FAA work together toward 
the common goal of safety and certification. 

• Recommendation R5.5: The FAA should emphasize that the ODA system should allow 
for direct contact between the E-UMs and the FAA technical experts without fear of 
reprisal for the ODA E-UMs. The FAA should also reinforce the need for the ODA to 
protect the E-UMs from reprisal so that the communication is as direct and as open as 
possible with the FAA technical staff. 

o Finding F5.5-A: There are a number of Boeing internal procedural layers that 
hinder the E-UMs from directly communicating with the BASOO/FAA engineers. 
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If an E-UM has questions or has difficulties with a subject, the E-UM is required 
to first try to solve the issue within the ODA instead of directly involving the 
BASOO experts. Although Boeing’s internal guidelines documented in D6- 
85963, Guidelines for Communicating Certification Related Issues, allow E-UMs 
to directly contact the FAA, this contact is for technical-only communication and 
only to better understand a documented FAA method of compliance. Even though 
the communication is allowed between the E-UM and FAA, the E-UM would 
have to submit a summary of the conversation to the ODA. These procedural 
layers may prevent “free” communication of issues/concerns to the FAA. 

• Recommendation R5.6: The FAA should review all oversight corrective actions and 
survey results (open and closed) raised by the BASOO to identify any systemic trends in 
non-compliances and ensure all open findings are being actioned in the appropriate 
timeframe. 

o Observation 05.6-A: There have been many Boeing ODA corrective actions 
initiated and verified by the BASOO since 2009. These corrective actions provide 
10 years of oversight findings on the Boeing ODA and are a valuable source of 
data to analyze and review the perfonnance of the Boeing ODA, including any 
themes of recurring findings or longstanding open corrective actions. 

• Recommendation R5.7: The FAA should require Boeing to submit compliance data 
recommending FAA approval for FAA flight test activities. Compliance data submissions 
should include FAA Form 8100-9, Statement of Compliance with Airworthiness 
Standards, signed by the appropriate E-UM recommending approval of the data. 

o Observation 05.7-A: For FAA flight test activities, the Boeing ODA manual 
specifically excludes E-UMs’ use of “Recommend Approval” using FAA Form 
8100-9 for FAA-retained flight test reports. This practice of the Boeing ODA not 
recommending approval of flight test data is inconsistent with other ODAs. The 
Boeing ODA manual states that Boeing will assist and submit the compliance data 
for approval; however, compliance data submitted by Boeing should be 
accompanied by FAA Fonn 8100-9(s) signed by the appropriate E-UM(s) 
recommending approval of these data. 

o Finding F5.7-A: The Boeing ODA manual refers to internal procedures 

documents, which the JATR team believes creates an additional and unnecessary 
level of complexity. This practice of referring to internal procedures is 
inconsistent with other ODAs, which have a single manual (standalone document) 
that contains all necessary information. 
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Integrated Approach to Development and Certification 
6. Holistic, integrated aircraft-level approach 


Recommendation R6 

Based on the JATR team’s observations and findings related to the design process of the 
flight control system and the related system safety assessments for the B 73 7 MAX, JATR 
team members recommend that the FAA promote a safety culture that drives a primary 
focus on the creation of safe products, which in turn comply with certification requirements. 
Aircraft functions should be assessed, not in an incremental and fragmented manner, but 
holistically at the aircraft level. System function and performance, including the effects of 
failures, should be demonstrated and associated assumptions should be challenged to 
ensure robust designs are realized. The safety analysis process should be integrated with the 
aircraft development assurance process to ensure all safety requirements and associated 
assumptions are correct, complete, and verified. The FAA should encourage applicants to 
have a system safety function that is independent from the design organization, with the 
authority to impartially assess aircraft safety and influence the aircraft/system design 
details. Adoption of a safety management system is one way this can be achieved. 


Recommendation R6 is based on the following observations, findings, and supporting 
recommendations related to the JATR team’s review of the design process of the flight control 
system and the related SSAs for the B737 MAX. In achieving R6, JATR team members advise 
the FAA to take actions that include, but are not necessarily limited to, the supporting 
recommendations below. 

• Recommendation R6.1: The FAA should ensure applicants improve adherence to fail¬ 
safe design concept principles when designing or modifying systems. The FAA should 
encourage applicants not to design only for compliance, but also to follow basic 
principles to design for safety when developing or changing system functions. This 
should include elimination of hazards and use of design features, warnings, and 
procedures. 

o Observation 06.1-A: Proper flight crew action was considered an adequate 
mitigation to risks such as erroneous activation of MCAS. 

o Finding F6.1-A: The JATR team identified that the design process was not 
sufficient to identify all the potential MCAS hazards. As part of the single¬ 
channel speed trim system, the MCAS function did not include fault tolerant 
features, such as sensors voting or limits of authority, to limit failure effects 
consistent with the hazard classification. 
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o Finding F6.1-B: The use of pilot action as a primary mitigation means for MCAS 
hazards, before considering eliminating such hazards or providing design features 
or warnings to mitigate them, is not in accordance with Boeing’s process 
instructions for safe design in the conception of MCAS for the B737 MAX. 

o Finding F6.1-C: The JATR team found that there was a missed opportunity to 
further improve the system design through the use of available fail-safe design 
principles and techniques presented in AC 25.1309-1A and in EASA AMC 
25.1309 in the MCAS design. 

• Recommendation R6.2: As part of the certification process for transport category 
airplanes, the FAA should examine all “major hazards” where a key mitigation is flight 
crew action to see if they are potentially catastrophic. The FAA should evaluate the 
impact of the hazard and its mitigations at the aircraft level, including the impact on the 
crew and cockpit environment, to detennine if additional mitigating design features are 
required. 

o This recommendation is based on Findings F6.1-A, F6.1-B, and F6.1-C, above. 

• Recommendation R6.3: The FAA should implement policies and further guidance to 
reinforce that all system functions that are used in flight critical functions should 
implement means for increased fault tolerance, such as signal health monitoring, voting 
means, and failure annunciation. Increased system fault tolerance should be sought to the 
extent practicable to accommodate unforeseen scenarios or unconfirmed assumptions 
during system operation. 

o This recommendation is based on Findings F6.1-A, F6.1-B, and F6.1-C, above. 

• Recommendation R6.4: The FAA should implement policies and further guidance to 
reinforce that workload evaluations should not be limited to the areas affected by the 
design changes alone. Workload evaluation should be perfonned with the complete flight 
deck effects of the failure conditions, including associated procedures. 

o Finding F6.4-A: When all flight deck effects are considered, the introduction of 
the MCAS function invalidated aircraft-level assumptions for flight crew 
responses related to erroneous AO A failures under certain conditions. A complete 
workload assessment was not perfonned for validation of the erroneous AOA 
effects with the added MCAS functionality. The same assumptions for flight crew 
responses to erroneous AOA were carried over from previous programs without 
formal validation. 
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• Recommendation R6.5: The FAA should emphasize the need to perform a functional 
SSA. The complete system function, including interfaces and unchanged parts of the 
implementation, should be assessed. When adding new functions, a complete top-down 
safety assessment process from the aircraft level should be performed. Special emphasis 
should be given on exercising care for reuse of safety assessment analysis infonnation. 

o Finding F6.5-A: An integrated SSA to investigate the MCAS as a complete 
function was not perfonned. The safety analyses were fragmented among several 
documents, and parts of the SSA from the B737 NG were reused in the B737 
MAX without sufficient evaluation. ARP4754A section 6, for example, has 
guidance that should have been used, given Boeing’s election to follow 
ARP4754A as means of compliance. 

o Observation 06.5-A: While the JATR team would not expect applicants to 
prepare a specific SSA for a flight control law, safety analyses should be 
conducted from a functional perspective. Applicants may document the safety 
analyses in multiple documents provided the documents are well organized and 
clearly record the results of the safety analyses conducted from a functional 
perspective. 

• Recommendation R6.6: The FAA should ensure that when new functions are introduced, 
the applicants develop a new FHA specific to that function that is used to develop design 
mitigations for identified hazards. 

o Finding F6.6-A: There is a perception that the FHA reports are not used to drive 
the design; rather, they are used to document the design as already defined. The 
STS and flight control computer (FCC) FHAs were updated reports from the 
B737 NG, and in the JATR team’s assessment, they did not appear to be used as 
tools to identify new hazards related to MCAS and drive design mitigations. As 
an example, in the hierarchy of safety solutions, mitigation by design should be 
prioritized over warnings and training/procedures. By documenting the as-is 
configuration, Boeing concluded that pilot training and procedures were sufficient 
to ensure safety. 

• Recommendation R6.7: The FAA should encourage applicants to have a system safety 
function that is independent from the design organization in order to independently assess 
aircraft safety, and that has the authority to influence the aircraft/system design details. 
Such system safety function should ensure that comprehensive and integrated risk, 
failure, and safety analyses are perfonned any time a design change is made that could 
affect the safe operation of the aircraft. Adoption of a safety management system is one 
way this can be achieved. 
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o Observation 06.7-A: The JATR team could not identify whether Boeing has an 
independent safety group coordinating the systems safety analysis and their role 
within a development program. Each system team conducts both the design and 
related safety assessments. It was also noted that flight test pilots, including the 
chief pilot, are often used to validate key design decisions. This is not a problem, 
as long as the engineers have robust infonnation, which might be an issue, 
considering that the system description and SSA documentation are fragmented. 

o Finding F6.7-A: The Boeing analysis of erroneous MCAS activation did not 
adequately take into account what else might be happening at the same time, such 
as the possibility of an AOA failure with all its associated flight deck effects 
potentially distracting the crew from recognizing the trim action. 

• Recommendation R6.8: Given the importance of the single & multiple failure (S&MF) 
analysis or equivalent in the development assurance process, the FAA should require the 
S&MF analysis or equivalent as a certification deliverable to demonstrate system-level 
integration and the effects of cascading hazards at the aircraft level. 

o Observation 06.8-A: The aircraft-level S&MF analysis was developed and used 
by Boeing as an internal document and not as a certification deliverable. The 
JATR team expects the S&MF analysis to have been a compliance artifact and 
provided to the FAA to demonstrate system-level integration. This was already an 
EASA recommendation in its certification review item CRI-F6 for the B737-8 
MAX. 

• Recommendation R6.9: The FAA should not accept analysis of a single “worst-case 
scenario” as covering all possible failure modes of the related systems. The FAA should 
require applicants to analyze each function to identify failure modes for each signal input 
considering all foreseeable scenarios and the multiple possible outcomes for each flight 
phase in their cascading effects analysis. 

o Observation 06.9-A: As far as AOA failures are concerned, the JATR team 

observed that the S&MF analysis was limited to a single worst-case scenario: loss 
of AOA on one side plus erroneous AOA on the other side. Other AOA failures 
were not evaluated because the worst-case failure modes were analyzed and the 
hazard analyses in the S&MF was limited to only those combined worst-case 
failures. Because this worst-case scenario was already considered catastrophic, 
the S&MF analysis was not updated after introduction of MCAS. 

o Finding F6.9-A: Evaluating worst-case scenario for the AOA failures was not 
adequate to identify the hazardous effects (including complete flight deck effects) 
of the single AOA failures. 
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o Observation 06.9-B: Boeing engineers did not see MCAS as “new or novel,” 
partly because it was already operational on the military tanker version of the 
B767. 

o Observation 06.9-C: Boeing conducted an S&MF analysis on Revision C of the 
STS requirements for MCAS software, which only included high-speed values in 
its lookup table (as was used in the military tanker version of the B767). 

o Observation 06.9-D: During Boeing flight tests, the company added low-speed 
values to the MCAS lookup table in its Revision D of the STS requirements for 
MCAS. 

o Observation 06.9-E: The B737-8 MAX was certified with Revision E of the STS 
requirements for MCAS software. 

o Observation 06.9-F: The SSA was not updated beyond Revision C of the STS 
requirements for MCAS. The JATR team observed no documented risk, failure, 
or safety analyses conducted on the MCAS software beyond Revision C. 

o Observation 06.9-G: Boeing detennined the high-speed regime to be the critical 
aspect of MCAS, and thus no revision to the SSA was necessary when the low 
speeds were added to the software’s lookup table. 

o Observation 06.9-H: Boeing concluded that multiple erroneous MCAS 
activations were not worse than a single erroneous activation, based on the 
assumption that the crew would return the aircraft to a trimmed state (consistent 
with AC 25-7C guidance) following each activation. 

• Recommendation R6.10: The FAA should not accept a mitigation for the single “worst- 
case scenario” as mitigating all possible scenarios. The FAA should ensure that 
mitigations are developed as appropriate for the multiple outcomes identified in the 
cascading effects analysis. 

o This recommendation is based on Observation 06.9-A and Finding F6.9-A, 
above. 

• Recommendation R6.11: The FAA should require applicants to develop an SSA process 
description to be followed by each system for consistency of methodology, use of 
guidance, and assumptions. 

o Observation 06.11-A: The JATR team was unable to identify a Boeing document 
that provides a consistent process and methodology for the SSA process to be 
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followed across each system. Report templates are provided, but templates are not 
sufficient to ensure uniformity of analysis. 

• Recommendation R6.12: The FAA should develop a practice of questioning the validity 
of assumptions made by the applicant and require substantive support for all such 
assumptions. 

o Finding F6.12-A: The JATR observed in Issue Paper G-l that Boeing’s rationale 
for exceptions from current amendments for the B737 MAX was focused on 
similarity with the B737 NG model and the risk of confusing the pilots by 
introducing differences between the two models (e.g., exceptions for § 25.1322). 
These approaches were driven by Boeing’s assumptions that the MAX is a 
replacement for the NG and that MAX pilots will be experienced NG pilots. 

These assumptions were not warranted, as demonstrated by airlines for which the 
MAX was the first B737 model to be purchased (e.g., Air Canada), and by new 
pilots entering service directly to the MAX (e.g., the First Officer on ET302). 

o Finding F6.12-B: Basic assumptions about trained and qualified flight crew 

response to malfunctions used in the design and certification of the B737-8 MAX 
did not appear to hold in the two accident cases, based on preliminary 
information. 

7. Human Factors 


Recommendation R 7 

Based on the JATR team’s observations and findings related to human factors-related 
issues in the certification process, JATR team members recommend that the FAA integrate 
and emphasize human factors and human system integration throughout its certification 
process. Human factors-relevant policies and guidance should be expanded and clarified, 
and compliance with such regulatory requirements as 14 CFR §§ 25.1302 (Installed 
Systems and Equipment for Use by the Flightcrew), 25.1309 (Equipment, Systems, and 
Installations), and 25.1322 (Flightcrew Alerting) should be thoroughly verified and 
documented. To enable the thorough analysis and verification of compliance, the FAA 
should expand its aircraft certification resources in human factors and in human system 
integration. 


Recommendation R7 is based on the JATR team’s identification of human factors-related issues 
in the certification process. These issues are reflected in multiple recommendations throughout 
the JATR’s submittal, including but not limited to Recommendations R1.3, R1.9, R2.1, R2.2, 
R2.4, R2.8, R2.9, R2.10, R3.8, R3.9, R3.10, R3.ll, R3.13, R3.14, R3.15, R3.17, R6.2, R6.4, 
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R9.1, R9.6, R9.7, and R11.2. In achieving R7, JATR team members advise the FAA to also take 
actions that include, but are not necessarily limited to, the additional supporting recommendation 
below. 


• Recommendation R7.1: The FAA should expand its aircraft certification resources in 
human factors and in human system integration to enable the thorough analysis and 
verification of compliance with such regulatory requirements as 14 CFR §§ 25.1302 
(Installed Systems and Equipment for Use by the Flightcrew) and 25.1322 (Flightcrew 
Alerting). 

o Observation 07.1-A: There are very few human factors specialists in the FAA’s 
nationwide Aircraft Certification Organization. 

o Observation 07.1-B: The FAA has extremely limited human factors and human 
system integration resources in an era where most safety failures are linked to 
human-machine interaction. 

• Recommendation R7.2: The FAA should review existing guidance material and update as 
necessary to emphasize the importance of human factors and human system integration 
throughout the certification process. 

o Observation 07.2-A: Existing human factors guidance material (e.g., AC 25- 
1302-1) may be insufficient to emphasize the importance of human factors and 
human system integration throughout the certification process. (See also 
Observations 02.1-A and 02.2-A and Findings F2.2-A and F6.4-A) 

8. Development assurance 


Recommendation R8 

Based on the JATR team’s observations and findings related to the development assurance 
process applied to the design of the flight control system of the B737 MAX, JATR team 
members recommend that the FAA ensure applicants apply industry best practice for 
development assurance, including requirements management, visibility of assumptions, 
process assurance activities, and configuration management. The FAA should ensure 
achievement of the close coupling that is required between the applicant safety analysis 
process and the development assurance process to classify failure conditions and derive the 
level of rigor of design development and verification. A current example of industry best 
practice is SAE International’s Aerospace Recommended Practice 4754 (ARP4754). 

The FAA should review and amend Advisory Circular 20-174 to clearly articulate the 
principles of ARP4754, promoting industry best practice for development assurance of 
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aircraft and aircraft systems to address applicants’ design trend of increasing integration 
between aircraft functions and systems. 


Recommendation R8 is based on the following observations, findings, and supporting 
recommendations related to the JATR team’s review of the Boeing development assurance 
process applied to the design of the flight control system of the B737 MAX. In achieving R8, 
JATR team members advise the FAA to take actions that include, but are not necessarily limited 
to, the supporting recommendations below. 

• Recommendation R8.1: The FAA, as part of the BASOO oversight activities, should 
review the Boeing development assurance process to ensure industry best practice for 
development assurance is being followed for an integrated approach to design changes at 
the aircraft, system, subsystem and item levels. Developing internal procedures to more 
robustly meet the objectives of ARP4754A and the adoption of Dynamic Object-Oriented 
Requirements System (DOORS) to manage all requirements is one way to achieve this 
integrated approach. 

o Observation 08.1-A: The Boeing development assurance process was agreed to 
by the FAA via Issue Paper SA-1; however, the process was not aligned to 
industry best practice for integrated systems. 

a) Section 3 of the B737 MAX Development Assurance Compliance Plan 
D925A003-01 states that the development assurance process meets the 
objectives of ARP4754A; however, the JATR found numerous instances 
where the development assurance process does not satisfy the objectives 
of ARP4754A for an integrated approach to design. 

b) Not all requirements are traceable, and assumptions are managed 
independent of requirements. 

c) Boeing attempted to apply ARP4754A methodology to the B737 MAX, 
however the benefits could not be fully realized as the application was 
limited to changed areas. The retirement of older design provides 
opportunities to use more fulsome assurance methods. 

o Observation 08.1-B: The JATR team’s assessment is that Boeing’s integration of 
the design and safety analysis is heavily reliant on the use of the chief pilot or test 
pilot to perform development assurance integration functions at the aircraft level. 

a) It is not clear to the JATR team how pilot verification and validation 
activities are captured within an integrated approach to design 
development. 
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b) It is not clear to the JATR team how discipline specialists are identified 
for assessment of a design change after regression testing has been 
completed, or how these specialists provide assistance to an aircraft-level 
integrated assessment. 

o Finding F8.1-A: The Boeing development assurance process, as applied to the 
B737 MAX, can be improved to more robustly meet the objectives of ARP4754A 
for an integrated approach to design - specifically, the integration of more 
complex systems and software into legacy aircraft. 

• Recommendation R8.2: The FAA, as part of the BASOO oversight activities, should 
review the Boeing safety analysis process, including how candidate items are identified 
for the S&MF analysis, to ensure hazards are assessed in an integrated manner across 
systems and subsystems, and all credible hazards are identified for assessment at the 
aircraft level. 

o Observation 08.2-A: The Boeing safety analysis process is not fully aligned with 
the development assurance process. 

a) Documenting identified risks and mitigating only the “worst-case scenario” 
does not necessarily identify all critical failure modes, particularly when the 
interaction between related systems is not considered. 

b) Requirements-based testing and intended-function testing may not 
adequately capture cascading failure conditions if the S&MF candidate item 
list does not adequately document a complete set of hazards. 

o Finding F8.2-A: The Boeing safety analysis process, as applied to the B737 
MAX, can be improved to be more integrated with the development assurance 
process at the aircraft, system, subsystem, and item levels. 

• Recommendation R8.3: The FAA, as part of the BASOO oversight activities, should 
review the Boeing safety analysis process and ensure it is aligned with the Boeing 
development assurance process to meet the objectives of ARP4754A. A more robust 
alignment between these two processes will ensure completeness of hazard identification 
in the S&MF candidate list, identification of all critical failure modes, and incorporation 
of the mitigations into the design. 

o This recommendation is based on Observation 08.2-A and Finding F8.2-A, 
above. 

• Recommendation R8.4: The FAA, as part of the BASOO oversight activities, should 
review the Boeing process for managing assumptions to ensure assumptions are visible 
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throughout the development assurance and safety analysis processes. Increased visibility 
includes the integrated reassessment of assumptions to ensure that associated hazards are 
appropriately identified and remain valid and that the design complies with functional 
and safety requirements derived from assumptions. 

o Observation 08.4-A: The JATR team’s assessment is that the approach taken to 
record assumptions in coordination sheets (and not DOORS) results in a loss of 
visibility of assumptions through the development assurance process. The 
assumptions listed in the coordination sheet are not directly visible to system- 
level requirements, and there is no obvious feedback loop to ensure these 
assumptions remain valid throughout the development process. 

• Recommendation R8.5: The FAA, as part of the BASOO oversight activities, should 
ensure Boeing implements a more iterative approach to verify and validate requirement 
functional dependencies and assess the interaction between hazards identified at the 
system level and the aircraft level. Such an approach would increase the involvement of 
system safety specialists, human factors specialists, and pilots to perform independent 
reviews of potential hazard impacts at the aircraft level. This independent review would 
supplement and infonn the aircraft-level development assurance integration activities 
carried out by the Boeing chief pilot/test pilot. 

o Finding F8.5-A: An opportunity exists for Boeing to adopt an integrated approach 
for requirements management though use of requirements management tools like 
DOORS for all requirements. This will improve the robustness of requirements 
management and verification and validation activities. 

a) While the JATR team observed that Boeing had managed requirements 
through a number of different processes, this does not meet the objectives of 
ARP4754A for an integrated approach to requirements management. 

b) The process assurance checklist for requirements identified as “Alternate 
MoC” and “Alternate MoC Plus” may not be sufficient to address the 
integration effects of the design change. 

c) Adopting an integrated approach using a requirements management tool like 
DOORS will allow airworthiness authorities and delegated persons/designees 
to easily and independently review findings of compliance and understand 
the interrelationships between systems to ensure completeness of certification 
activities. 
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o Finding F8.5-B: Establishing requirements baselines at system and subsystem 
levels will assist configuration management of requirements throughout the 
development assurance process. 

a) The process to establish a requirements baseline should be aligned to the 
configuration management process. 

b) At a minimum, the establishment of baselines for requirements identified as 
“safety” will facilitate increased control of these requirements throughout the 
development assurance process. 

o Observation 08.5-A: DOORS information notes observed by the JATR team read 
like requirements and were reportedly identified in the coordination sheet but 
were not identified in DOORS as a requirement to be verified. 

• Recommendation R8.6: The FAA, as part of the BASOO oversight activities, should 
ensure Boeing improves the system architecture used for requirements management. This 
includes expanding the use of a requirements management tool, such as DOORS, to 
manage ah requirements to improve the integration of system- and item- level 
requirements to other systems and items and also to parent aircraft-level requirements. 

o This recommendation is based on Observation 08.5-A and Findings F8.5-A and 
F8.5-B, above. 

• Recommendation R8.7: To the extent applicants rely on original aircraft- and system- 
level assumptions, the FAA should ensure the applicants perform a thorough review of 
system design changes to ensure they are not inconsistent with those assumptions. 

o This recommendation is based on Finding F6.4-A, above. 

• Recommendation R8.8: The FAA should emphasize in guidance that, besides 
requirements-based testing, the applicant should perform robustness test cases for 
identifying and investigating unexpected system effects and flight crew responses. For 
example, the process should account for evaluation of cases where pilots do not follow 
the assumptions (e.g., not trimming out the failure). 

o This recommendation is based on Finding F6.4-A, above. 

• Recommendation R8.9: The FAA should develop, validate, and implement design and 
analysis models, methodologies, and approaches capable of identifying interactions 
among systems such as the catastrophic interaction between the AOA system and MCAS. 

o Observation 08.9-A: FAA Order 8110.48A, How to Establish the Certification 
Basis for Changed Aeronautical Products, provides the following guidance in 
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paragraph 2-1: “Essentially, a substantial design change is an alteration to a 
product that is so extensive that the design models, methodologies, and 
approaches used to demonstrate a previous compliance finding cannot be used.” 

o Finding F8.9-A: The B737-8 MAX accident scenarios were not identified during 
the testing and certification process. This is an indication that the “design models, 
methodologies, and approaches” used to demonstrate compliance need 
improvement to identify interactions among systems. 

• Recommendation R8.10: The FAA should review AC 20-174 to ensure that expectations 
for a holistic aircraft-level design assurance practice for transport category aircraft is 
achieved which includes consideration of all systems (including safety) requirements and 
assumptions. In particular, the AC should address how credit can be given for traditional 
techniques for simple detenninistic systems within a structured methodology. 

o Finding F8.10-A: AC 20-174 does not provide clear and unambiguous guidance 
for the application of ARP4754A to Part 25 Aircraft. 

o Observation O8.10-A: AC 20-174 provides for limited application of the 

development assurance process “where traditional techniques have been shown to 
be acceptable for more traditional systems designs.” 

a) Not requiring the more structured techniques be applied as indicated in AC 
20-174 may result in misinterpretation that the structured methodology would 
not be required. This is not the expectation of the ARP4754A. 

b) There is no definition provided within AC 20-174 to identify what constitutes 
a “traditional systems design” or “traditional techniques.” 

c) Previous design practice considered as “traditional” did not include the design 
assurance processes rigor provided by ARP4754A. The process is expected to 
identify, validate, and verity all system requirements including safety 
requirements and would include identification and disposition of all 
assumptions. 

• Recommendation R8.11: The FAA should ensure applicants provide a full list of all 
aircraft proposed changes (no matter how trivial), complete with a system description and 
all interfaces associated with each proposed change, such that an infonned assessment 
can be made using established criteria prior to agreeing on the systems which will be 
subject to limited application of a development assurance process. 

o Finding F8.11-A: The practice of applying a limited application of a development 
assurance process for modifications to aircraft or systems can be improved - 
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specifically, the criteria used to assess each proposed modification and the 
requirement to satisfy safety assessment objectives. 

o Observation 08.11-A: The limited application of a development assurance 
process agreed between the FAA and Boeing did not adequately establish the 
criteria for determining which new or modified systems require certification 
compliance findings relative to development assurance. 

a) Each candidate system should be critically assessed against a robust set of 
criteria. 

b) Criteria should be informed by the objectives and requirements of 
ARP4754A. 

c) The FAA should be provided with sufficient insight into the modifications to 
make an informed assessment of each proposed modification against the 
established criteria. 

d) The rationale and decisions resulting from this assessment should be 
documented. 

• Recommendation R8.12: The FAA should ensure that agreement of any limited 
application of a development assurance process includes the requirement for the 
applicant’s safety analysis processes to satisfy the ARP 4754A safety assessment 
objectives. 

o Observation 08.12-A: The limited application of a development assurance 
process agreed between the FAA and Boeing did not adequately consider the 
applicant’s safety analysis process and how that integrates with the tailored 
development assurance process for complex and integrated systems. 

a) The FAA’s participation in system reviews did not result in ensuring Boeing’s 
process was equivalent to ARP4754A. 

b) The expectation that safety requirements be considered within the design 
assurance process was not realized. 

c) ARP4754A Section 6 provides the necessary guidance for modifications to 
aircraft or systems. 

d) ARP4754A Section 5.1 details the objectives of the safety assessment process 
regarding analysis of functional interactions and interdependencies. 
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Impact of Design Changes on Operations and Training 
9. Impact of Product Design Changes on Operations 


Recommendation R9 

Based on the JATR team’s findings and observations related to the operational design 
assumptions of crew response applied during the certification process for the flight control 
system of the B737 MAX, JATR team members recommend that the FAA require the 
integration of certification and operational functions during the certification process. The 
FAA should be provided all system differences between related aircraft in order to 
adequately evaluate operational impact, systems integration, and human performance. 


Recommendation R9 is based on the following observations, findings, and supporting 
recommendations related to the JATR team’s review of the operational design assumptions of 
crew response that Boeing applied during the certification process for the B737 MAX. In 
achieving R9, JATR team members advise the FAA to take actions that include, but are not 
necessarily limited to, the supporting recommendations below. 

• Recommendation R9.1: The FAA should revise AC 120-53B and FAA Order 8900.1 
Volume 8, Chapter 2 to include an assessment of the cumulative effects of changed 
products, such as differences in aircraft systems, displays, flight characteristics, and 
procedures. 

o Observation 09.1-A: AC 120-53B does not require the cumulative effects on 
system changes to be considered. 

o Observation 09.1-B: Boeing submitted to the FAA’s AEG a list of features of the 
B7378 MAX cockpit which were changed from the base model B737-800. In 
Issue Paper 0-1, Type Rating Determination and 14 CFR Training Requirements, 
the FAA raised concerns about cumulative effects of system changes from the 
B737 NG to the B737 MAX that may cause greater than level B differences 
training. Boeing’s response to this concern was that there was no precedent in 
prior Boeing amended type certification projects and that AC 120-53B did not 
require the cumulative effects on system changes to be considered. The FAA 
accepted Boeing’s response on 26 January 2016. 

• Recommendation R9.2: The FAA should review and if necessary revise AC 120-53B to 
ensure that the AEG and FSB are provided with all the system differences between 
related aircraft irrespective of engineering determination of the safety significance. 
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o Observation 09.2-A: Issue Paper 0-6 and FAA Order 8110.4C articulate AEG’s 
responsibility, among other things, to address Flight Standards considerations 
such as contribution of operational perspective to engineering activities during the 
type certification process. The Order specifically requires AEG’s early 
involvement in the certification process starting at the requirements definition 
phase of the system’s life-cycle. 

o Finding F9.2-A: The limited infonnation provided to the FSB limited their ability 
to assess the operational impacts of failures of systems associated with MCAS 
and the subsequent requirements for flight crew training. With the infonnation 
AEG was provided, it is reasonable to conclude that the FSB would not know the 
full impact of the changed design and thus would be unaware that they had been 
provided insufficient infonnation to adequately comply with the requirements in 
FAA Order 8110.4C. 

• Recommendation R9.3: Where the assessment of the effectiveness of differences training 
is not conducted in an aircraft, the FAA should require the AEG to use operational flight 
crew complements (e.g., line captains and line first officers), with a range of flight 
experience, as part of the assessment. 

o Observation 09.3-A: To be consistent with §§ 25.671 and 25.672, and to comply 
with the guidance in AC 25-7C, Boeing utilized four fundamental assumptions on 
crew actions in the flight control FHA for the B737 MAX and other Boeing 
models. The third assumption, taken from AC 25-7C, stated: “The pilot will take 
immediate action to reduce or eliminate high control forces by re-trimming or 
changing configuration or flight conditions.” 

o Finding F9.3-A: Based on the JATR team’s review of preliminary accident 
infonnation, in both aircraft accidents the flight crew did not appear to meet the 
“immediate action” assumption. This assumption makes no allowance for 
differing training and certification requirements for flight crew operating under 
other CAAs. The FAA requires an air transport license with 1,500 hours 
experience before being employed by a Part 121 operator. Other CAAs have no 
such requirement, with co-pilots required only to have a commercial pilot’s 
license. 

• Recommendation R9.4: The AEG should have deeper involvement during the 
certification process and collaborate closely with FAA’s Aircraft Certification 
Service (AIR) to ensure they have the proper knowledge to make informed decisions 
about operational suitability issues that may be affected by certification details. 
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o Observation 09.4-A: Pilots working in the certification process may not have 
complete knowledge of operational issues, while pilots working in the operational 
evaluation process may not have complete knowledge of certification issues. This 
may contribute to a lack of communication between the two processes. 

o Finding F9.4-A: Communication of MCAS functionality between certification 
and AEG was not sufficiently robust for AEG to fully understand MCAS 
implications in an operational environment. 

• Recommendation R9.5: The FAA should conduct a study to determine the value of AEG 
pilots receiving familiarization training to enhance their understanding of certification 
flight tests. 

o This recommendation is based on Observation 09.4-A and Finding F9.4-A, 
above. 

• Recommendation R9.6: The FAA should review and if necessary revise AC 25.1302-1, 
Installed Systems and Equipment for Use by the Flightcrew, to ensure that failures of 
related systems are assessed taking into account human performance and the operational 
environment utilizing an AEG operational specialist. 

o Observation 09.6-A: A review of preliminary accident reports 

KNKT.18.10.35.04 and AI-0/19 indicates that the complex operational 
environment that faced the flight crews and the associated workload may not have 
been anticipated in the certification process. 

o Finding F9.6-A: AC 25.1302-1 does not adequately address the operational aspect 
of an aircraft’s design. 

o Finding F9.6-B: AC 25.1302-1, paragraph l-2(a), Applicability, lists a number of 
certification roles that the guidance is directed toward, and the list does not 
include an operational pilot specialist such as an aviation safety inspector from the 
AEG. 

• Recommendation R9.7: The FAA should review and if necessary revise guidance 
material to ensure that operational considerations associated with the design change are 
adequately risk-assessed to minimise the potential for flight crew error. 

o This recommendation is based on Observation 09.6-A and Findings F9.6-A and 
F9.6-B, above. 
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10. Impact of product design changes on flight crew training 


Recommendation RIO 

Based on the JATR team s findings and observations related to flight crew training, JATR 
team members recommend that the FAA require a documented process to determine what 
information will be included in the Airplane Flight Manual, the Flight Crew Operating 
Manual, and the Flight Crew Training Manual. The FAA should review training programs 
to ensure flight crews are competent in the handling of mis-trim events. 


Recommendation RIO is based on the following observations, findings, and supporting 
recommendations related to the JATR team’s review of the FCOM, FCTM, and AFM developed 
during the certification process for the B7378 MAX. In achieving RIO, JATR team members 
advise the FAA to take actions that include, but are not necessarily limited to, the supporting 
recommendations below. 

• Recommendation RIO. 1: The FAA should include in the FSB report the flight experience 
level and qualification of the flight crew used to assess the effectiveness of the 
differences training. 

o Finding F10.1-A: Boeing’s test pilots are not a representative sample of the 
operators’ pilot population. 

o Finding F10.1-B: The AEG pilots are not a representative sample of the 
operators’ pilot population. 

o Finding F10.1-C: A validation flight conducted under AC 120.53B is conducted 
with an experienced line captain in the left seat, and includes an experienced 
flight test pilot in the right seat as a safety pilot. 

o Finding F10.1-D: The AEG’s evaluation flights do not evaluate crew performance 
and do not represent operators’ pilots’ experience level or operation. 

o This recommendation is also based on Observation 09.3-A and Finding F9.3-A, 
above. 

• Recommendation R10.2: The FAA should review the B737 MAX type rating training 
program to include training in the operation of the manual stabilizer trim wheel 
throughout the speed range. 
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o Observation O10.2-A: A review of preliminary accident reports 

KNKT.18.10.35.04 and AI-0/19 indicates both flights suffered an extreme mis- 
trim event which involved the activation of the MCAS function. 

o Finding F10.2-A: A review of the B737 MAX type rating training syllabus 

indicates that, although exercises are conducted in the flight simulator to address a 
STAB TRIM runaway, the syllabus does not specifically address awareness of 
airspeed versus the forces required to manually trim the aircraft and to recognize 
and correct a mis-trim state. 

• Recommendation R10.3: The FAA should require operators of the B737 to include 
operation of the manual stabilizer trim wheel throughout the speed range in their 
recurrent training programs. 

o This recommendation is based on Observation O10.2-A and Finding F10.2-A, 
above. 

• Recommendation R10.4: The FAA should add a special emphasis training item to the 
B737 FSB Report to include training in the operation of the main electric stabilizer trim 
and the manual stabilizer trim wheel and recovery from a mis-trim state throughout the 
speed range. 

o This recommendation is based on Observation O10.2-A and Finding F10.2-A, 
above. 

• Recommendation RIO.5: The FAA should develop a documented process to detennine 
what information will be included in the AFM, FCOM, and FCTM. The process must 
include agreement from all disciplines (e.g., certification, operations, maintenance, 
human factors) for the system or function descriptions to be removed. 

o Observation O10.5-A: Information related to the MCAS functionality within the 
FCC originally was in the draft FCOM and was subsequently removed (around 
the time of MCAS Revision D, in early 2016), but without a fonnal process in 
place to ensure agreement from all disciplines on the removal of that information. 
Technology, even if it functions without pilot involvement, may be integrated 
with other aircraft systems. One system or functional failure could impact other 
systems requiring pilot involvement. 

o Finding F10.5-A: Information related to MCAS functionality and failure scenarios 
is critical for pilot knowledge and understanding of the system as it interfaces 
with the aircraft’s trim system and AO A inputs. 
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• RIO is also supported by Recommendation R3.14 and accompanying Finding F3.14-A, 
and by Recommendation R9.2 and accompanying Observation 09.2-A and Finding F9.2- 
A, above. 

11. Impact of product design changes on maintenance training 


Recommendation Rll 

JATR team members recommend that the FAA conduct a study to determine the adequacy 
of policy, guidance, and assumptions related to maintenance and ground handling training 
requirements. 


In furtherance of R11, JATR team members advise the FAA to take actions that include, but are 
not necessarily limited to, the supporting recommendations below. 

• Recommendation R11.1: The FAA should conduct a study to focus on adequacy of 
maintenance and ground handling differences training requirements for transport 
category aircraft. 

o Observation 011.1 -A: JATR tasking included assessing the adequacy of 
policy/guidance and assumptions related to training of mechanics and ground 
handlers for new and related aircraft. The B737 MAX Maintenance Review Board 
Chairman briefed the JATR team. The JATR team solicited an AEG maintenance 
specialist that was not associated with the B737 MAX certification activities for 
discussion. 

o Finding FI 1.1-A: The JATR team was unable to make a determination of the 
adequacy of policy/guidance and assumptions related to training of mechanics and 
ground handlers for new/related aircraft. 

• Recommendation R11.2: The FAA should develop regulatory requirements to consider 
and mitigate potential errors by maintenance technicians and by ground handling 
personnel as part of the certification process of a product. 

o Observation 011.2-A: Section 25.1302 requires applicants to consider and mitigate 
potential flight crew errors. 

o Observation 011.2-B: There are no aircraft-level regulatory requirements, equivalent 
to § 25.1302, to consider and mitigate potential errors by maintenance technicians or 
by ground handling personnel. 
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o Observation Oil .2-C: Maintenance and ground handling errors have contributed to 
several accidents and multiple incidents, and maintenance issues might also be 
relevant to the Lion Air B737 MAX accident based on the preliminary report. 

Post-Certification Activities 

12. Post-Certification Corrective Actions and Data Sharing 


Recommendation R12 

JATR team members recommend that the FAA review its policies for analyzing safety risk 
and implementing interim airworthiness directive action following a fatal transport aircraft 
accident. The FAA should ensure that it shares post-accident safety information with the 
international community to the maximum extent possible. 


Recommendation R12 is based on the following observations, findings, and supporting 
recommendations. In achieving R12, JATR team members advise the FAA to take actions that 
include, but are not necessarily limited to, the supporting recommendations below. 

• Recommendation R12.1: The FAA should review FAA Order 8110.107A, Monitor 
Safety/Analyze Data, and consider reducing the control program risk guideline for post¬ 
accident corrective action if a catastrophic fatal accident of a transport category aircraft 
has occurred. For example, the allowable FAA Monitor Safety/Analyze Data (MSAD) 
guidelines for control program fleet risk for the related corrective action could be reduced 
to between 10% and 25% of their normal values. 

o Observation 012.1-A: The FAA uses the MSAD process, Order 8110.107A, to 
manage potential safety issues. If an unsafe condition is discovered, the FAA uses 
the MSAD process to assess the adequacy of the timeline to implement a 
corrective action (the airworthiness directive (AD) compliance time) using 
quantitative risk analysis. The FAA compares the calculated control program fleet 
and individual risk to an allowable risk guideline. 

o Observation 012.1-B: Most of the safety issues assessed and managed using the 
MSAD process are triggered by in-service data, production escapes, or 
engineering discoveries. Actions taken in response to these precursors, through 
use of the MSAD process, generally proactively prevent a fatal accident from 
occurring. 

o Finding F12.1-A: It is evident that the impact of a second fatal transport aircraft 
accident due to the same cause far exceeds the impact of the first. When the 
MSAD process is initiated due to a fatal transport accident, there should be lower 
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risk tolerance for a second accident and the control program fleet risk guideline 
should be reduced. 

• Recommendation R12.2: The FAA, in hannonization with other CAAs, should review 
the airworthiness directive processes to detennine the need and proper intervals for a 
flight crew pre-flight briefing when an interim action AD mandates an existing AFM 
procedure or mandates a revision to the AFM to address a major contributing factor to a 
catastrophic fatal accident of a transport category aircraft. 

o Observation 012.2-A: After a catastrophic fatal accident, an interim corrective 
action is often issued to prevent a second accident. Sometimes flight crew 
procedural changes are used, and the interim action AD requires a revision to the 
AFM. 

o Finding F12.2-A: Flight crew procedural changes can be ineffective. After the 
Helios Airways Flight 522 accident in 2005, the FAA issued AD 2006-13-13 that 
required a flight crew recall item: if the altitude warning hom sounds, don the 
oxygen masks. After issuance of the AD, an FAA inspector was performing an 
enroute inspection of a major U.S. airline B737 flight crew when the altitude 
warning horn sounded. Despite the AFM revision, the crew did not don their 
masks as required. 12 

o Finding F12.2-B: A method to increase the effectiveness of a flight crew 
procedural change is to require that the AFM revision be part of a pre-flight 
briefing. Having briefed a procedure, the crew is more likely to remember and 
perform the procedure correctly if the need arises during flight. This briefing 
could be performed less frequently than every flight, for example before the first 
flight of the day or other suitable interval. 

• Recommendation R12.3: Where the FAA assigns responsibility for continued operational 
safety oversight of a product to a different FAA office than the one that conducted 
oversight of the type certification, the agency should ensure that it has sufficient 
mechanisms in place for the transfer of requisite technical knowledge about the design to 
the responsible office. 

o Observation 012.3-A: The FAA BASOO is responsible for overseeing the Boeing 
ODA and certification of Boeing products, while the Seattle Aircraft Certification 
Office (SACO) is responsible for overseeing continued operational safety 
management of Boeing products once they are certificated. This transfer of 

12 After the FAA issued AD 2006-13-13, it received continuing reports of in-service events involving failure of the 
flight crew to recognize and react properly to valid cabin altitude warning horns. Therefore, the FAA issued AD 
2008-23-07 that required a new flight crew briefing before the first flight of the day and following any change in 
flight crewmembers, in addition to the existing AFM procedures. 
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responsibility after the product is certificated was not a review area of the JATR 
team, but the team assumes the transfer may involve familiarizing and/or briefing 
SACO staff on design details. 

o Observation 012.3-B: Some details, such as the system safety analyses related to 
the MCAS function, were fragmented among several documents for the B737 
MAX. This could hinder the successful transfer of infonnation to the SACO for 
the purpose of overseeing the continued operational safety management of the 
product. 

• Recommendation R12.4: The FAA should review its safety infonnation sharing policy to 
ensure that it shares technical safety information with other CAAs to the maximum extent 
possible. Maximum sharing of such infonnation would enhance safety and minimize 
inconect speculation by parties that are not participants in an ongoing accident 
investigation. 

o Observation 012.4-A: As a participant in an accident investigation conducted by 
another State, the FAA is obligated under International Civil Aviation 
Organization (ICAO) Annex 13, Aircraft Accident and Incident Investigation, not 
to divulge infonnation on the progress and the findings of the investigation 
without the express consent of the State conducting the investigation. 

o Observation 012.4-B: A benefit of participating in an accident investigation as 
the State of Design is that it allows risks to be addressed by the State of Design as 
quickly as possible. 13 Unfortunately, due to constraints on the flow of information 
from the design state and other CAAs, the third-party CAAs are reliant on the 
State of Design to act in their interest. The State of Design will apply their own 
risk processes as influenced by their regulatory and cultural environment. Delays 
or absence of authoritative and consistent communication results in the outside 
party’s speculation. 

o Observation 012.4-C: ICAO Annex 13 also recommends that, “States should 
promote the establishment of safety infonnation sharing networks among all users 
of the aviation system and should facilitate the free exchange of infonnation on 
actual and potential safety deficiencies.” 

o Finding F12.4-A: Restrictions on the flow of safety information impacted the 
capacity and efficiency of the JATR under the umbrella of, “the accidents are still 
under investigation.” 


13 Part 21 defines “State of Design’’ as “the country or jurisdiction having regulatory authority over the organization 
responsible for the design and continued airworthiness of a civil aeronautical product or article.” 14 CFR 21.1(b)(8). 
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